• Pipeline overview
  • Dataset generation
    • Outcome variable: Binarizing and recoding details
  • Data exploration and visualization
  • Robust feature evaluation
    • Feature importances: Random Forest
      • Mean decrease in impurity (MDI)
      • Permutation importance
      • MDI vs Permutation importance
    • Feature importances: LASSO
      • Coefficient-based variable importance
      • Permutation importance
      • Coefficient vs. Permutation importance
  • Generation of final model
    • Performance
    • Features: permutation importance
    • Comparison with bootstrap results
    • Features: permutation importance
    • Comparison with bootstrap results
  • Other cleanup

Project run parameters

no_tasks=2
seed = 9384
outcome = 'i_drug'
fbase = '/scratch/p_gaydosh_lab/DSI/'
results_directory <- str_c(fbase, outcome, '/results_full_h2o_2_deleted')
set.seed(seed)

Purpose. In this work, we will explore the relation between identified measures of despair of interest (e.g., personality measures of self-consciousness, individual and composite item scores from the CES-D assessment) and descriptors of diseases of despair. We will achieve this goal through modeling the outcomes based on the included predictors, and robustly assess the importance of the included features in predicting the outcomes via bootstrapping. We will use two well-known machine learning models, random forests and LASSO, which are both frequently used to measure the relative importance of the predictors included in the models. Lastly, we’ll generate trained and tuned models using this reduced feature set which can be used by others wish to predict the identified outcomes.

Subject inclusion. For this investigation, we will omit the entirety of Wave 2. This is commonly done in analyses of AddHealth data due the design of the original study. Otherwise, our dataset will include only subjects who have predictor and outcome data in all of the waves.

Outcome variables. In this experiment, we assess suicidal ideation at Wave 5.

Predictor variables. The predictors for these models are hand-picked, and based on previous work, relevance, and subject matter expertise. The set of predictors and the set of outcomes are disjoint. Predictors from Waves 1-4 (excluding Wave 2, see above) are included, and will be detailed in the following analysis.

Pipeline overview

Dataset generation

The predictors we will be using will be the the variable predictor_list loaded from 10-import-data.Rmd file. These initial set of predictors will be based of the list of variables that describe anxiety, depression, and optimism.

## set outcome variable of interest
filebase = '/scratch/p_gaydosh_lab'

#create data in specified form
dataset_list <- generate_datasets(outcome, binarize=FALSE, filebase=filebase, seed_val=seed)
Parsed with column specification:
cols(
  variable = col_character(),
  wave = col_double(),
  na_level = col_double(),
  type = col_character()
)
UNRELIABLE VALUE: Future (‘<none>’) unexpectedly generated random numbers without specifying argument '[future.]seed'. There is a risk that those random numbers are not statistically sound and the overall results might be invalid. To fix this, specify argument '[future.]seed', e.g. 'seed=TRUE'. This ensures that proper, parallel-safe random numbers are produced via the L'Ecuyer-CMRG method. To disable this check, use [future].seed=NULL, or set option 'future.rng.onMisuse' to "ignore".
UNRELIABLE VALUE: Future (‘<none>’) unexpectedly generated random numbers without specifying argument '[future.]seed'. There is a risk that those random numbers are not statistically sound and the overall results might be invalid. To fix this, specify argument '[future.]seed', e.g. 'seed=TRUE'. This ensures that proper, parallel-safe random numbers are produced via the L'Ecuyer-CMRG method. To disable this check, use [future].seed=NULL, or set option 'future.rng.onMisuse' to "ignore".
UNRELIABLE VALUE: Future (‘<none>’) unexpectedly generated random numbers without specifying argument '[future.]seed'. There is a risk that those random numbers are not statistically sound and the overall results might be invalid. To fix this, specify argument '[future.]seed', e.g. 'seed=TRUE'. This ensures that proper, parallel-safe random numbers are produced via the L'Ecuyer-CMRG method. To disable this check, use [future].seed=NULL, or set option 'future.rng.onMisuse' to "ignore".
UNRELIABLE VALUE: Future (‘<none>’) unexpectedly generated random numbers without specifying argument '[future.]seed'. There is a risk that those random numbers are not statistically sound and the overall results might be invalid. To fix this, specify argument '[future.]seed', e.g. 'seed=TRUE'. This ensures that proper, parallel-safe random numbers are produced via the L'Ecuyer-CMRG method. To disable this check, use [future].seed=NULL, or set option 'future.rng.onMisuse' to "ignore".
UNRELIABLE VALUE: Future (‘<none>’) unexpectedly generated random numbers without specifying argument '[future.]seed'. There is a risk that those random numbers are not statistically sound and the overall results might be invalid. To fix this, specify argument '[future.]seed', e.g. 'seed=TRUE'. This ensures that proper, parallel-safe random numbers are produced via the L'Ecuyer-CMRG method. To disable this check, use [future].seed=NULL, or set option 'future.rng.onMisuse' to "ignore".0 subjects removed from dataset.
[1] "Recoding Missing Factor Variables"
[1] "Number of factor variables being recoded :  98"
[1] "Factor variables being recoded : "
 [1] "h1fs1"   "h1fs3"   "h1fs4"   "h1fs5"   "h1fs6"   "h1fs7"   "h1fs11"  "h1fs15"  "h1fs16"  "h1fs17" 
[11] "h1fs19"  "h1ee14"  "h1ee12"  "h1ds14"  "h1ed9"   "h1ed7"   "h1ds3"   "h1fv7"   "h1fv1"   "h1fv8"  
[21] "h1jo9"   "h1ds13"  "h1ds12"  "h1ds11"  "h3sp5"   "h3sp6"   "h3sp7"   "h3sp8"   "h3sp9"   "h3sp10" 
[31] "h3sp11"  "h3sp12"  "h3sp13"  "h3ec56"  "h3ds18h" "h3ds18a" "h3ds18i" "h3to49"  "h3ds6"   "h3ds5"  
[41] "h3ds4"   "h4id5h"  "h4mh18"  "h4mh19"  "h4mh20"  "h4mh21"  "h4mh22"  "h4mh23"  "h4mh24"  "h4mh25" 
[51] "h4mh26"  "h4mh27"  "h4id5j"  "h4pe6"   "h4pe14"  "h4pe22"  "h4pe30"  "h4pe7"   "h4pe15"  "h4pe23" 
[61] "h4pe31"  "h4mh3"   "h4mh4"   "h4mh5"   "h4mh6"   "h4mh2"   "hdl"     "ldl"     "tg"      "h4bpcls"
[71] "h4ds7"   "h4ds19"  "h4ds14"  "h4ds20"  "h4ds6"   "h4ds5"   "h4ds4"   "h4cj17"  "h5id6g"  "h5ss0a" 
[81] "h5ss0b"  "h5ss0c"  "h5ss0d"  "h5ss0e"  "h5id6i"  "h5pe1"   "h5pe2"   "h5pe3"   "h5mn1"   "h5mn2"  
[91] "h5mn3"   "h5mn4"   "h5cj1d"  "h5cj1e"  "h5cj1f"  "h5cj1b"  "h5cj1c"  "h5cj1a" 
[1] "h1fs1"
[1] "h1fs3"
[1] "h1fs4"
[1] "h1fs5"
[1] "h1fs6"
[1] "h1fs7"
[1] "h1fs11"
[1] "h1fs15"
[1] "h1fs16"
[1] "h1fs17"
[1] "h1fs19"
[1] "h1ee14"
[1] "h1ee12"
[1] "h1ds14"
[1] "h1ed9"
[1] "h1ed7"
[1] "h1ds3"
[1] "h1fv7"
[1] "h1fv1"
[1] "h1fv8"
[1] "h1jo9"
[1] "h1ds13"
[1] "h1ds12"
[1] "h1ds11"
[1] "h3sp5"
[1] "h3sp6"
[1] "h3sp7"
[1] "h3sp8"
[1] "h3sp9"
[1] "h3sp10"
[1] "h3sp11"
[1] "h3sp12"
[1] "h3sp13"
[1] "h3ec56"
[1] "h3ds18h"
[1] "h3ds18a"
[1] "h3ds18i"
[1] "h3to49"
[1] "h3ds6"
[1] "h3ds5"
[1] "h3ds4"
[1] "h4id5h"
[1] "h4mh18"
[1] "h4mh19"
[1] "h4mh20"
[1] "h4mh21"
[1] "h4mh22"
[1] "h4mh23"
[1] "h4mh24"
[1] "h4mh25"
[1] "h4mh26"
[1] "h4mh27"
[1] "h4id5j"
[1] "h4pe6"
[1] "h4pe14"
[1] "h4pe22"
[1] "h4pe30"
[1] "h4pe7"
[1] "h4pe15"
[1] "h4pe23"
[1] "h4pe31"
[1] "h4mh3"
[1] "h4mh4"
[1] "h4mh5"
[1] "h4mh6"
[1] "h4mh2"
[1] "hdl"
[1] "ldl"
[1] "tg"
[1] "h4bpcls"
[1] "h4ds7"
[1] "h4ds19"
[1] "h4ds14"
[1] "h4ds20"
[1] "h4ds6"
[1] "h4ds5"
[1] "h4ds4"
[1] "h4cj17"
[1] "h5id6g"
[1] "h5ss0a"
[1] "h5ss0b"
[1] "h5ss0c"
[1] "h5ss0d"
[1] "h5ss0e"
[1] "h5id6i"
[1] "h5pe1"
[1] "h5pe2"
[1] "h5pe3"
[1] "h5mn1"
[1] "h5mn2"
[1] "h5mn3"
[1] "h5mn4"
[1] "h5cj1d"
[1] "h5cj1e"
[1] "h5cj1f"
[1] "h5cj1b"
[1] "h5cj1c"
[1] "h5cj1a"
   h1fs1    h1fs3    h1fs4    h1fs5    h1fs6    h1fs7   h1fs11   h1fs15   h1fs16   h1fs17   h1fs19   h1ee14 
"factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" 
  h1ee12   h1ds14    h1ed9    h1ed7    h1ds3    h1fv7    h1fv1    h1fv8    h1jo9   h1ds13   h1ds12   h1ds11 
"factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" 
   h3sp5    h3sp6    h3sp7    h3sp8    h3sp9   h3sp10   h3sp11   h3sp12   h3sp13   h3ec56  h3ds18h  h3ds18a 
"factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" 
 h3ds18i   h3to49    h3ds6    h3ds5    h3ds4   h4id5h   h4mh18   h4mh19   h4mh20   h4mh21   h4mh22   h4mh23 
"factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" 
  h4mh24   h4mh25   h4mh26   h4mh27   h4id5j    h4pe6   h4pe14   h4pe22   h4pe30    h4pe7   h4pe15   h4pe23 
"factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" 
  h4pe31    h4mh3    h4mh4    h4mh5    h4mh6    h4mh2      hdl      ldl       tg  h4bpcls    h4ds7   h4ds19 
"factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" 
  h4ds14   h4ds20    h4ds6    h4ds5    h4ds4   h4cj17   h5id6g   h5ss0a   h5ss0b   h5ss0c   h5ss0d   h5ss0e 
"factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" 
  h5id6i    h5pe1    h5pe2    h5pe3    h5mn1    h5mn2    h5mn3    h5mn4   h5cj1d   h5cj1e   h5cj1f   h5cj1b 
"factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" "factor" 
  h5cj1c   h5cj1a 
"factor" "factor" 
[1] "Recoding Missing Numeric Variables"
[1] "Number of numeric variables being recoded :  7"
[1] "Numeric variables being recoded : "
[1] "h1ed2"   "h4bmi"   "hba1c"   "crp"     "h4waist" "h4sbp"   "h4dbp"  
#parse out dataset components
wave_data <- dataset_list$wave_data
full_dataset <- dataset_list$full_dataset
ds_raw <- dataset_list$ds_raw_outcome
ds <- dataset_list$ds_final

#ml splits of the data
training_df <- dataset_list$training_df
validation_df <- dataset_list$validation_df
testing_df <- dataset_list$testing_df

Outcome variable: Binarizing and recoding details

The following table details the values present in the outcome variable. As we can see, we need to convert the variable into a factor, and drop the NAs.

ds_raw %>% 
  group_by(.data[[outcome]]) %>% 
  summarise(total = n(), type = class(.data[[outcome]]))
ABCDEFGHIJ0123456789
i_drug
<dbl>
total
<int>
type
<chr>
08930numeric
1324numeric
NA95numeric
3 rows

ds %>% 
  group_by(.data[[outcome]]) %>% 
  summarise(total = n(), type = class(.data[[outcome]]))
ABCDEFGHIJ0123456789
i_drug
<fctr>
total
<int>
type
<chr>
08930factor
1324factor
2 rows

print(str_c('Total number of subjects in unprocessed outcome data: ', nrow(ds_raw)))
[1] "Total number of subjects in unprocessed outcome data: 9349"
print(str_c('Total number of subjects in processed outcome data: ', nrow(ds)))
[1] "Total number of subjects in processed outcome data: 9254"
print(str_c('Total subjects dropped due to NA or skip: ', nrow(ds_raw)-nrow(ds)))
[1] "Total subjects dropped due to NA or skip: 95"

The table also demonstrates that the classes are very imbalanced, with about 14x the negative class as compared with the positive class. This is displayed graphically for easy consumption below.

ds_raw %>%
  explore_outcome(ds, outcome)
The original raw outcome data is displayed by converting it to a factor.  For more information about its actual type, investigate the ds_raw dataframe.  It is most likely a double.

After dropping the NAs, we see that we now have 9168 rows, which is consistent with the table above. The distribution of the data has a distinct imbalance as noted above. I think this warrants using pr_auc as the optimization and selection metric.

Data exploration and visualization

Here, we comment about the general characteristics of the data based on the provided visualizations. We comment on missingness of data, any strange or unusual behavior (e.g., strong imbalances), and any correlation that sticks out.

#Report about the characteristics of the subjects left out of the join
ds %>% explore_dropped()


# Visualize distributions of variables of interest
ds %>% 
  dplyr::select(-aid) %>%
  graph_bar_discrete(df = .,
                     plot_title = "Distributions of Discrete Variables",
                     max_categories = 50,
                     num_rows = 3,
                     num_cols = 3,
                     x_axis_size = 12,
                     y_axis_size = 12,
                     title_size = 15)

ds %>%
  graph_missing(only_missing = TRUE,
                title = "Percent Missing",
                box_line_size = .5,
                label_size = .5,
                x_axis_size = 12,
                y_axis_size = 12,
                title_size = 15)


ds %>%
  #dplyr::select(1:20) %>%
  pairwise_cramers_v() %>%
  plot_cramer_v(x_axis_angle = 90,
                plot_title = "Association among Categorical Variables",
                interactive = TRUE)
h4id5hh5id6gh1fs1h1fs3h1fs4h1fs5h1fs6h1fs7h1fs11h1fs15h1fs16h1fs17h1fs19h3sp5h3sp6h3sp7h3sp8h3sp9h3sp10h3sp11h3sp12h3sp13h4mh18h4mh19h4mh20h4mh21h4mh22h4mh23h4mh24h4mh25h4mh26h4mh27h5ss0ah5ss0bh5ss0ch5ss0dh5ss0eh5id6ih4id5jh4pe6h4pe14h4pe22h4pe30h4pe7h4pe15h4pe23h4pe31h5pe1h5pe2h5pe3h4mh3h4mh4h4mh5h4mh6h4mh2h5mn1h5mn2h5mn3h5mn4h1ee14h1ee12h3ec56hdlldltgh4bpclsh1ds14h1ed9h1ed7h1ds3h1fv7h1fv1h1fv8h1jo9h1ds13h1ds12h1ds11h3ds18hh3ds18ah3ds18ih3to49h3ds6h3ds5h3ds4h4ds7h4ds19h4ds14h4ds20h4ds6h4ds5h4ds4h4cj17h5cj1dh5cj1eh5cj1fh5cj1bh5cj1ch5cj1ai_drugh4id5hh5id6gh1fs1h1fs3h1fs4h1fs5h1fs6h1fs7h1fs11h1fs15h1fs16h1fs17h1fs19h3sp5h3sp6h3sp7h3sp8h3sp9h3sp10h3sp11h3sp12h3sp13h4mh18h4mh19h4mh20h4mh21h4mh22h4mh23h4mh24h4mh25h4mh26h4mh27h5ss0ah5ss0bh5ss0ch5ss0dh5ss0eh5id6ih4id5jh4pe6h4pe14h4pe22h4pe30h4pe7h4pe15h4pe23h4pe31h5pe1h5pe2h5pe3h4mh3h4mh4h4mh5h4mh6h4mh2h5mn1h5mn2h5mn3h5mn4h1ee14h1ee12h3ec56hdlldltgh4bpclsh1ds14h1ed9h1ed7h1ds3h1fv7h1fv1h1fv8h1jo9h1ds13h1ds12h1ds11h3ds18hh3ds18ah3ds18ih3to49h3ds6h3ds5h3ds4h4ds7h4ds19h4ds14h4ds20h4ds6h4ds5h4ds4h4cj17h5cj1dh5cj1eh5cj1fh5cj1bh5cj1c
Association among Categorical Variables

The correlation plot suggests that there are several variables we might think about removing. Firstly, there are moderate correlations among the individual predictor blocks; this is due to the way that they are ordered, since they’re essentially grouped into subsets. Addtionally, some predictor pairs have extremely high correlations, like (h4id5j, h4mh26) (correlation of 0.72), and many variables in the h4mh* series. (h3id5j, h4id5h) also has high correlation > 0.5. We may want to consider removing several of these in addition to the age variables because it may cause feature importance masking within our approaches.

Robust feature evaluation

The following table displays the mean performance metrics for the bootstrapped models on the validation set, removing values for which there are NA.


mean_bs_rf_perf <- bs_rf_perf %>%
  summarise_if(is.numeric, mean, na.rm=TRUE) %>%
  mutate(model = 'bs_rf') %>%
  dplyr::select(model, everything())

mean_bs_rf_perf
ABCDEFGHIJ0123456789
model
<chr>
accuracy
<dbl>
mpce
<dbl>
sens
<dbl>
spec
<dbl>
ppv
<dbl>
npv
<dbl>
roc_auc
<dbl>
pr_auc
<dbl>
tns
<dbl>
bs_rf0.96474590.49220640.0168750.99871220.31978220.96592890.81162130.17325771783.7
1 row | 1-10 of 27 columns

As shown, the bootstrapped models tend to have high specificity but low sensitivity, indicating that there is a challenge in identifying subjects with suicidal ideation.

Feature importances: Random Forest

Mean decrease in impurity (MDI)

boot_rf_mdi <- list(bs_rf_mdi) %>%
  get_median_placement(use_base_var = TRUE) %>%
  add_attribute_names('predictor', full_dataset) %>%
  dplyr::select(predictor, att_name, overall_rank)

head(boot_rf_mdi, 20)
ABCDEFGHIJ0123456789
predictor
<chr>
att_name
<chr>
overall_rank
<int>
h4dbpS27 DIASTOLIC BLOOD PRESSURE -W41
age_w5age_w52
h4sbpS27 SYSTOLIC BLOOD PRESSURE-W43
h4bmiS27 BMI -W44
h4waistS27 MEASURED WAIST (CM) -W45
h5cj1cS14Q1C 12 MO SELL DRUGS-W56
crpcrp7
one_race5one_race58
h1ed2S5Q2 FREQ-SKIPPED SCHOOL-W19
h3bmih3bmi10
Next
12
Previous
1-10 of 20 rows

This table returns the MDI variable importance ranks that returned from each of the bootstrapped models.

# Needs to be fixed so that axes don't overlap each other and obscure understanding
plot_placement_boxplot(list(bs_rf_mdi))

Permutation importance

Now, let’s look at the permutation importance:

met <- 'pr_auc'
bs_rf_perm <- bs_rf_perm_plt %>%
  get_permute_placement(metric_oi=met) %>%
  add_attribute_names('predictor', full_dataset) %>%
  dplyr::select(predictor, everything())

head(bs_rf_perm, 20)
ABCDEFGHIJ0123456789
predictor
<chr>
att_name
<chr>
overall_rank
<int>
pr_auc
<dbl>
h5cj1cS14Q1C 12 MO SELL DRUGS-W510.029647167
h4ds5S21Q5 12 MO,OFT SELL DRUGS-W420.010081669
h4cj17S22Q17 EVER INCARCERATED-W430.009305191
h3ds5S26Q5 12 MO,OFT SELL DRUGS-W340.005107186
h5ss0bS10Q0B PAST 7 DAY DEPRESSED-W550.004402274
h4ds6S21Q6 12 MO,OFT STEAL SOMETHING/<$50-W460.004022309
h3to49S28Q49 SINCE JUN95 HAVE DRIVEN DRUNK-W370.003096702
h5cj1dS14Q1D 12 MO GET IN PHYS FIGHT-W580.003067237
h5ss0aS10Q0A PAST 7 DAY SHAKE BLUE-W590.002639547
h4ds7S21Q7 12 MO,OFT PART PHYS FIGHT/GRP-W4100.002618647
Next
12
Previous
1-10 of 20 rows

MDI vs Permutation importance

In this step, we assess the differences generated between the different types of importances.

cbind(boot_rf_mdi[1:20,], dplyr::select(bs_rf_perm[1:20,], -all_of(met)))
ABCDEFGHIJ0123456789
predictor
<chr>
att_name
<chr>
overall_rank
<int>
predictor
<chr>
h4dbpS27 DIASTOLIC BLOOD PRESSURE -W41h5cj1c
age_w5age_w52h4ds5
h4sbpS27 SYSTOLIC BLOOD PRESSURE-W43h4cj17
h4bmiS27 BMI -W44h3ds5
h4waistS27 MEASURED WAIST (CM) -W45h5ss0b
h5cj1cS14Q1C 12 MO SELL DRUGS-W56h4ds6
crpcrp7h3to49
one_race5one_race58h5cj1d
h1ed2S5Q2 FREQ-SKIPPED SCHOOL-W19h5ss0a
h3bmih3bmi10h4ds7
Next
12
Previous
1-10 of 20 rows | 1-4 of 6 columns

As shown, the MDI importance suffers from imbalances due to the number of values associated with a predictor. Because the wave ages have so many more values than the other factors, this artificially inflates their importance in MDI. The permutation importance is more intuitive.

plot_permute_var_imp(bs_rf_perm, metric = met)

In this step, we model the relation between the outcomes and the predictors using a linear regression with L2 regularization. This drives the importance of unimportant and redudant features towards zero.

mean_bs_lasso_perf <- bs_lasso_perf %>%
  summarise_if(is.numeric, mean, na.rm=TRUE) %>% 
  mutate(model='bs_lasso') %>%
  dplyr::select(model, everything())

mean_bs_lasso_perf
ABCDEFGHIJ0123456789
model
<chr>
accuracy
<dbl>
mpce
<dbl>
sens
<dbl>
spec
<dbl>
ppv
<dbl>
npv
<dbl>
roc_auc
<dbl>
pr_auc
<dbl>
tns
<dbl>
bs_lasso0.94340540.41015680.210.96968650.20205150.97164460.7221350.13715211731.86
1 row | 1-10 of 27 columns

Feature importances: LASSO

Coefficient-based variable importance

boot_lasso_mdi <- list(bs_lasso_mdi) %>%
  get_median_placement(use_base_var = TRUE) %>%
  add_attribute_names('predictor', full_dataset) %>%
  dplyr::select(predictor, att_name, overall_rank)

head(boot_lasso_mdi, 20)
ABCDEFGHIJ0123456789
predictor
<chr>
att_name
<chr>
overall_rank
<int>
one_race5one_race51
age_w5age_w52
bio_sexBIOLOGICAL SEX-W13
h4cj17S22Q17 EVER INCARCERATED-W44
h4dbpS27 DIASTOLIC BLOOD PRESSURE -W45
h1ed2S5Q2 FREQ-SKIPPED SCHOOL-W16
h4bmiS27 BMI -W47
h1ds3S29Q3 LIE TO PARENTS ABOUT WHEREABOUT-W18
h3sp7S12Q7 PAST 7 DAYS FELT AS GOOD AS OTH-W39
h5cj2bS14Q2B 12 MO SAW ONE SHOOT/STAB PERS-W510
Next
12
Previous
1-10 of 20 rows
plot_placement_boxplot(list(bs_lasso_mdi))

Permutation importance

bs_lasso_perm <- bs_lasso_perm_plt %>%
  get_permute_placement(metric_oi=met) %>% #set in random forest section
  add_attribute_names('predictor', full_dataset) %>%
  dplyr::select(predictor, everything())

head(bs_lasso_perm, 20)
ABCDEFGHIJ0123456789
predictor
<chr>
att_name
<chr>
overall_rank
<int>
pr_auc
<dbl>
h5cj1cS14Q1C 12 MO SELL DRUGS-W510.054165390
h4ds5S21Q5 12 MO,OFT SELL DRUGS-W420.012534262
h4mh2S14Q2 HOW OFTEN FEEL ISOLATED-W430.011379094
h4mh24S14Q24 PAST 7 DAYS FELT HAPPY-W440.011105397
h4ds6S21Q6 12 MO,OFT STEAL SOMETHING/<$50-W450.010869625
one_race5one_race560.009682284
h5pe1S9Q1 ALWAYS OPTIM ABOUT FUT-W570.009617185
h5mn3S13Q3 LAST MO FELT THINGS GO YOUR WAY-W580.009388152
h3to49S28Q49 SINCE JUN95 HAVE DRIVEN DRUNK-W390.008662670
hdlhdl100.008567205
Next
12
Previous
1-10 of 20 rows
plot_permute_var_imp(bs_lasso_perm, metric = met)

Coefficient vs. Permutation importance

Now, we compare the feature importances generated by the two different approaches. The traditional method of evaluating feature importance for regression methods is through analysis of the coefficients.

cbind(boot_lasso_mdi[1:20,], dplyr::select(bs_lasso_perm[1:20,], -met))
ABCDEFGHIJ0123456789
predictor
<chr>
att_name
<chr>
overall_rank
<int>
predictor
<chr>
one_race5one_race51h5cj1c
age_w5age_w52h4ds5
bio_sexBIOLOGICAL SEX-W13h4mh2
h4cj17S22Q17 EVER INCARCERATED-W44h4mh24
h4dbpS27 DIASTOLIC BLOOD PRESSURE -W45h4ds6
h1ed2S5Q2 FREQ-SKIPPED SCHOOL-W16one_race5
h4bmiS27 BMI -W47h5pe1
h1ds3S29Q3 LIE TO PARENTS ABOUT WHEREABOUT-W18h5mn3
h3sp7S12Q7 PAST 7 DAYS FELT AS GOOD AS OTH-W39h3to49
h5cj2bS14Q2B 12 MO SAW ONE SHOOT/STAB PERS-W510hdl
Next
12
Previous
1-10 of 20 rows | 1-4 of 6 columns

The following table compares the mean performance of bootstrapped random forests to the mean performance of bootstrapped LASSO methods.

bs_comp_perfs <- bind_rows(mean_bs_rf_perf, mean_bs_lasso_perf) 
bs_comp_perfs
ABCDEFGHIJ0123456789
model
<chr>
accuracy
<dbl>
mpce
<dbl>
sens
<dbl>
spec
<dbl>
bs_rf0.96474590.49220640.0168750.9987122
bs_lasso0.94340540.41015680.2100000.9696865
2 rows | 1-5 of 27 columns

Here, we look at the aggregated results of the bootstrapped predictors and compare the models generated to each other.

joined_results <- bs_rf_perm %>%
  dplyr::select(-met) %>%
  full_join(dplyr::select(bs_lasso_perm, -met), by=c("predictor", "att_name"), suffix=c('.rf', '.lasso')) %>%
  mutate(mean_rank = (overall_rank.rf+overall_rank.lasso)/2) %>%
  arrange(mean_rank)

head(joined_results, 20)
ABCDEFGHIJ0123456789
predictor
<chr>
att_name
<chr>
overall_rank.rf
<int>
overall_rank.lasso
<int>
mean_rank
<dbl>
h5cj1cS14Q1C 12 MO SELL DRUGS-W5111.0
h4ds5S21Q5 12 MO,OFT SELL DRUGS-W4222.0
h4ds6S21Q6 12 MO,OFT STEAL SOMETHING/<$50-W4655.5
h3to49S28Q49 SINCE JUN95 HAVE DRIVEN DRUNK-W3798.0
h4mh2S14Q2 HOW OFTEN FEEL ISOLATED-W41438.5
one_race5one_race51269.0
h5cj1dS14Q1D 12 MO GET IN PHYS FIGHT-W58119.5
h3ds5S26Q5 12 MO,OFT SELL DRUGS-W341610.0
h5mn3S13Q3 LAST MO FELT THINGS GO YOUR WAY-W515811.5
h5pe1S9Q1 ALWAYS OPTIM ABOUT FUT-W516711.5
Next
12
Previous
1-10 of 20 rows

The following visualization provides the intuition about the differences in the rankings between model types. They’re ordered by the overall mean importance, and for a given variable, the differences in rank are shown.

# Comparison of top_n features
joined_results %>%
  compare_feature_select(interactive = TRUE,
                         top_n = 100,
                         opacity = 0.50,
                         plot_title = "Permutation Importance of Predictors by Model")
0306090S14Q1E 12 MO PULL KNIFE/GUN SOMEONE-W5S14Q1F 12 MO SHOT/STABBED SOMEONE-W5S10Q19 LIFE NOT WORTH LIVING-W1S21Q20 12 MO,SHOT/STABBED SOMEONE-W4S21Q14 12 MO,SAW ONE SHOOT/STAB PERS-W4h3bmiS14Q22 PAST 7 DAYS FELT DEPRESSED-W4S14Q1A 12 MO DAMAGE PROP/NOT YOUR-W5S31Q1 SAW SHOOTING/STABBING OF PERSON-W1S14Q19 PAST 7 DAYS SHAKE OFF BLUES-W4S14Q2B 12 MO SAW ONE SHOOT/STAB PERS-W5S21Q19 12 MO,PULL KNIFE/GUN SOMEONE-W4S5Q6G EVER BEEN DX WITH DEPRESSION-W5S26Q15 DO NOT EXPECT THINGS MY WAY-W4S06Q5J EVER BEEN DX PANIC/ANXIETY DIS-W4S15Q56 CHANCE OF-LIVING TILL AGE 35-W3S12Q13 PAST 7 DAYS PEOPLE DISLIKED U-W3S14Q26 PAST 7 DAYS FELT SAD-W4S14Q5 LST MNTH THINGS GOING YOUR WAY-W4S5Q6I EVER BEEN DX PANIC/ANXIETY DIS-W5S30Q9 DRIVE WHILE DRUNK-W1S10Q4 JUST AS GOOD AS OTHER PEOPLE-W1S14Q21 PAST 7 DAYS TROUBL CONCENTRTNG-W4S38Q14 CHANCES-KILLED BY AGE 21-W1S12Q5 PAST 7 DAYS BOTHERED BY THINGS-W3S26Q6 12 MO,OFT STEAL SOMETHING/<$50-W3S12Q9 PAST 7 DAYS WERE DEPRESSED-W3S14Q27 PAST 7 DAYS FELT DISLIKED-W4S26Q6 WORRY ABOUT THINGS-W4S29Q13 STEAL WORTH < $50-W1S27 SYSTOLIC BLOOD PRESSURE-W4S10Q3 HAD THE BLUES-W1S29Q14 TAKE PART IN A GROUP FIGHT-W1hba1cS10Q11 HAPPY-W1S10Q7 TOO TIRED TO DO THINGS-W1S5Q7 RECEIVED OUT-OF-SCHL SUSPENSION-W1S14Q25 PAST 7 DAYS ENJOYED LIFE-W4S14Q3 LST MNTH UNABLE CONTROL THINGS-W4S26Q30 DO NOT WORRY ABOUT PAST-W4S10Q6 FELT DEPRESSED-W1S10Q16 FELT SAD-W1S10Q1 BOTHERED BY THINGS-W1S12Q8 PAST 7 DAYS TROUBLE CONCENTRTNG-W3S26Q31 RARELY EXPECT GOOD HAPPN TO ME-W4S12Q11 PAST 7 DAYS ENJOYED LIFE-W3S9Q3 EXPECT MORE GOOD-W5S27 MEASURED WAIST (CM) -W4S10Q0C PAST 7 DAY HAPPY-W5S26Q22 I GET STRESSED OUT EASILY-W4S29Q12 SELL DRUGS-W1S26Q7 ALWAYS OPTIMISTIC ABOUT FUTURE-W4crpS10Q0D PAST 7 DAY SAD-W5S38Q12 CHANCES-LIVE TO AGE 35-W1S10Q15 ENJOYED LIFE-W1S12Q12 PAST 7 DAYS WERE SAD-W3S27 BLOOD PRESSURE CLASS -W4S12Q1 PAST 7 DAYS SHAKE OFF BLUES-W3S10Q17 FELT PEOPLE DISLIKE YOU-W1S26Q23 EXPECT MORE GOOD THAN BAD-W4S13Q4 LAST MO DIFFS OVERWHELM-W5S12Q10 PAST 7 DAYS TOO TIRED DO THNGS-W3S14Q4 LST MNTH CONF HANDLE PERS PROB-W4S12Q7 PAST 7 DAYS FELT AS GOOD AS OTH-W3S14Q23 PAST 7 DAYS FELT TOO TIRED-W4S9Q2 NOT EXPECT THINGS MY WAY-W5S10Q5 TROUBLE KEEPING MIND FOCUSED-W1BIOLOGICAL SEX-W1S14Q24 PAST 7 DAYS FELT HAPPY-W4S14Q6 LST MNTH DIFFICULTIES OVERWHELM-W4S26Q14 NOT EASILY BOTHERED BY THINGS-W4S14Q1B 12 MO STEAL SOMETHING/>$50-W5S21Q7 12 MO,OFT PART PHYS FIGHT/GRP-W4S13Q1 LAST MO NO CNTRL IMPORT THINGS-W5tgS10Q0A PAST 7 DAY SHAKE BLUE-W5S5Q2 FREQ-SKIPPED SCHOOL-W1S14Q18 PAST 7 DAYS BOTHERED BY THINGS-W4S10Q0E PAST 7 DAY NOT WORTH-W5S14Q20 PAST 7 DAYS FELT AS GOOD OTHRS-W4S13Q2 LAST MO CONFID HANDLE PERS PBMS-W5ldlS27 BMI -W4S29Q3 LIE TO PARENTS ABOUT WHEREABOUT-W1age_w5S27 DIASTOLIC BLOOD PRESSURE -W4S22Q17 EVER INCARCERATED-W4hdlS10Q0B PAST 7 DAY DEPRESSED-W5S13Q3 LAST MO FELT THINGS GO YOUR WAY-W5S9Q1 ALWAYS OPTIM ABOUT FUT-W5S26Q5 12 MO,OFT SELL DRUGS-W3S14Q1D 12 MO GET IN PHYS FIGHT-W5one_race5S14Q2 HOW OFTEN FEEL ISOLATED-W4S28Q49 SINCE JUN95 HAVE DRIVEN DRUNK-W3S21Q6 12 MO,OFT STEAL SOMETHING/<$50-W4S21Q5 12 MO,OFT SELL DRUGS-W4S14Q1C 12 MO SELL DRUGS-W5
overall_rank.lassooverall_rank.rfPermutation Importance of Predictors by ModelPlacementsPredictorsmodel

Generation of final model

In this step, we build the final model for the random forest. We use slightly more values in order to come up with the best model, keeping in mind the number of combinations that are required to run to evaluate the grid.


final_rf_perf = NULL
  
# read file array
final_rf_perfs = load_from_csv(final_rf_perf, results_directory, no_tasks)

# get the index of the best performance and best performance
final_rfmodel_ind <- which.max(dplyr::select(final_rf_perfs, met) %>% pull(met))
final_rf_perf <- final_rf_perfs %>% slice(final_rfmodel_ind)

# load the model of interest
final_model_rf <- load_best_model('final_rf_model', results_directory, final_rfmodel_ind)

Performance

The final random forest performance metrics are shown below:

# show model final performance
print(final_rf_perf)
ABCDEFGHIJ0123456789
accuracy
<dbl>
kap
<dbl>
sens
<dbl>
spec
<dbl>
ppv
<dbl>
npv
<dbl>
mcc
<dbl>
j_index
<dbl>
bal_accuracy
<dbl>
0.93729730.27862640.406250.9563270.250.9782360.28766840.3625770.6812885
1 row | 1-9 of 26 columns

Features: permutation importance


final_rf_perm_plt = NULL
  
# read best perm plt
final_rf_perm_plt = load_from_csv(final_rf_perm_plt, results_directory, best_ind=final_rfmodel_ind)

final_rf_perm <- final_rf_perm_plt %>%
  get_permute_placement(metric_oi=met) %>%
  add_attribute_names('predictor', full_dataset) %>%
  dplyr::select(predictor, everything())

head(final_rf_perm, 20)
ABCDEFGHIJ0123456789
predictor
<chr>
att_name
<chr>
overall_rank
<int>
pr_auc
<dbl>
h4cj17S22Q17 EVER INCARCERATED-W410.06523748
h5ss0bS10Q0B PAST 7 DAY DEPRESSED-W520.04289668
h5ss0aS10Q0A PAST 7 DAY SHAKE BLUE-W530.04287412
h3to49S28Q49 SINCE JUN95 HAVE DRIVEN DRUNK-W340.03083555
h4mh2S14Q2 HOW OFTEN FEEL ISOLATED-W450.02819290
h5ss0cS10Q0C PAST 7 DAY HAPPY-W560.02400390
h5mn3S13Q3 LAST MO FELT THINGS GO YOUR WAY-W570.02234086
h1ed7S5Q7 RECEIVED OUT-OF-SCHL SUSPENSION-W180.02232680
h4mh27S14Q27 PAST 7 DAYS FELT DISLIKED-W490.02160583
h4dbpS27 DIASTOLIC BLOOD PRESSURE -W4100.02159901
Next
12
Previous
1-10 of 20 rows
plot_permute_var_imp(final_rf_perm, metric = met)

Comparison with bootstrap results

This section investigates the differences in the bootstrap results vs the features generated from the random forest final model. The following table shows the overall differences in rank.

rf_joined_results <- final_rf_perm %>%
  dplyr::select(-met) %>%
  full_join(dplyr::select(bs_rf_perm, -met), by=c("predictor", "att_name"), suffix=c('.final', '.bootstrap')) %>%
  mutate(mean_rank = (overall_rank.final + overall_rank.bootstrap)/2) %>%
  arrange(mean_rank)

head(rf_joined_results, 20)
ABCDEFGHIJ0123456789
predictor
<chr>
att_name
<chr>
overall_rank.final
<int>
overall_rank.bootstrap
<int>
h4cj17S22Q17 EVER INCARCERATED-W413
h5ss0bS10Q0B PAST 7 DAY DEPRESSED-W525
h3to49S28Q49 SINCE JUN95 HAVE DRIVEN DRUNK-W347
h5ss0aS10Q0A PAST 7 DAY SHAKE BLUE-W539
h4mh2S14Q2 HOW OFTEN FEEL ISOLATED-W4514
h5mn3S13Q3 LAST MO FELT THINGS GO YOUR WAY-W5715
h3ds5S26Q5 12 MO,OFT SELL DRUGS-W3184
h4dbpS27 DIASTOLIC BLOOD PRESSURE -W41013
h4ds6S21Q6 12 MO,OFT STEAL SOMETHING/<$50-W4226
h5mn2S13Q2 LAST MO CONFID HANDLE PERS PBMS-W51317
Next
12
Previous
1-10 of 20 rows | 1-4 of 5 columns

The following plot provides visualizations for the difference in the final model rankings vs the bootstrap.

# Comparison of top_n features
rf_joined_results %>%
  compare_feature_select(sel_cols = c("overall_rank.final", "overall_rank.bootstrap"),
    interactive = TRUE,
    top_n = 100,
    opacity = 0.50,
    plot_title = "Permutation Importance of Predictors: Final vs. Bootstrap")
0306090S26Q18A 12 MO,SAW ONE SHOOT/STAB PERS-W3S14Q1E 12 MO PULL KNIFE/GUN SOMEONE-W5S14Q1F 12 MO SHOT/STABBED SOMEONE-W5S06Q5H EVER BEEN DX WITH DEPRESSION-W4S14Q2B 12 MO SAW ONE SHOOT/STAB PERS-W5S06Q5J EVER BEEN DX PANIC/ANXIETY DIS-W4S10Q19 LIFE NOT WORTH LIVING-W1S21Q14 12 MO,SAW ONE SHOOT/STAB PERS-W4S31Q1 SAW SHOOTING/STABBING OF PERSON-W1S14Q1A 12 MO DAMAGE PROP/NOT YOUR-W5S14Q22 PAST 7 DAYS FELT DEPRESSED-W4S14Q19 PAST 7 DAYS SHAKE OFF BLUES-W4hba1cS26Q15 DO NOT EXPECT THINGS MY WAY-W4S29Q13 STEAL WORTH < $50-W1S10Q3 HAD THE BLUES-W1S15Q56 CHANCE OF-LIVING TILL AGE 35-W3h3bmiS12Q13 PAST 7 DAYS PEOPLE DISLIKED U-W3S26Q6 WORRY ABOUT THINGS-W4S14Q26 PAST 7 DAYS FELT SAD-W4S10Q16 FELT SAD-W1S12Q5 PAST 7 DAYS BOTHERED BY THINGS-W3S26Q22 I GET STRESSED OUT EASILY-W4S29Q14 TAKE PART IN A GROUP FIGHT-W1S14Q5 LST MNTH THINGS GOING YOUR WAY-W4S10Q17 FELT PEOPLE DISLIKE YOU-W1S14Q3 LST MNTH UNABLE CONTROL THINGS-W4S26Q7 ALWAYS OPTIMISTIC ABOUT FUTURE-W4S10Q11 HAPPY-W1crpS14Q1B 12 MO STEAL SOMETHING/>$50-W5S26Q6 12 MO,OFT STEAL SOMETHING/<$50-W3S38Q12 CHANCES-LIVE TO AGE 35-W1S10Q4 JUST AS GOOD AS OTHER PEOPLE-W1S12Q12 PAST 7 DAYS WERE SAD-W3S38Q14 CHANCES-KILLED BY AGE 21-W1S26Q14 NOT EASILY BOTHERED BY THINGS-W4S14Q21 PAST 7 DAYS TROUBL CONCENTRTNG-W4S14Q25 PAST 7 DAYS ENJOYED LIFE-W4S21Q5 12 MO,OFT SELL DRUGS-W4S26Q23 EXPECT MORE GOOD THAN BAD-W4S26Q31 RARELY EXPECT GOOD HAPPN TO ME-W4S9Q3 EXPECT MORE GOOD-W5S5Q6I EVER BEEN DX PANIC/ANXIETY DIS-W5S26Q30 DO NOT WORRY ABOUT PAST-W4S27 SYSTOLIC BLOOD PRESSURE-W4S12Q1 PAST 7 DAYS SHAKE OFF BLUES-W3S12Q7 PAST 7 DAYS FELT AS GOOD AS OTH-W3S14Q23 PAST 7 DAYS FELT TOO TIRED-W4S29Q12 SELL DRUGS-W1S10Q6 FELT DEPRESSED-W1S12Q8 PAST 7 DAYS TROUBLE CONCENTRTNG-W3S12Q9 PAST 7 DAYS WERE DEPRESSED-W3S14Q24 PAST 7 DAYS FELT HAPPY-W4S14Q1C 12 MO SELL DRUGS-W5S5Q6G EVER BEEN DX WITH DEPRESSION-W5S5Q2 FREQ-SKIPPED SCHOOL-W1S5Q7 RECEIVED OUT-OF-SCHL SUSPENSION-W1S9Q2 NOT EXPECT THINGS MY WAY-W5ldlS10Q1 BOTHERED BY THINGS-W1S10Q7 TOO TIRED TO DO THINGS-W1S12Q11 PAST 7 DAYS ENJOYED LIFE-W3S10Q15 ENJOYED LIFE-W1S21Q7 12 MO,OFT PART PHYS FIGHT/GRP-W4S14Q6 LST MNTH DIFFICULTIES OVERWHELM-W4S27 BLOOD PRESSURE CLASS -W4hdlS13Q1 LAST MO NO CNTRL IMPORT THINGS-W5S10Q5 TROUBLE KEEPING MIND FOCUSED-W1S13Q4 LAST MO DIFFS OVERWHELM-W5S14Q18 PAST 7 DAYS BOTHERED BY THINGS-W4S27 BMI -W4tgS30Q9 DRIVE WHILE DRUNK-W1S14Q20 PAST 7 DAYS FELT AS GOOD OTHRS-W4S14Q1D 12 MO GET IN PHYS FIGHT-W5S29Q3 LIE TO PARENTS ABOUT WHEREABOUT-W1S12Q10 PAST 7 DAYS TOO TIRED DO THNGS-W3S9Q1 ALWAYS OPTIM ABOUT FUT-W5S10Q0D PAST 7 DAY SAD-W5S10Q0E PAST 7 DAY NOT WORTH-W5S14Q27 PAST 7 DAYS FELT DISLIKED-W4age_w5BIOLOGICAL SEX-W1S27 MEASURED WAIST (CM) -W4one_race5S14Q4 LST MNTH CONF HANDLE PERS PROB-W4S10Q0C PAST 7 DAY HAPPY-W5S13Q2 LAST MO CONFID HANDLE PERS PBMS-W5S21Q6 12 MO,OFT STEAL SOMETHING/<$50-W4S27 DIASTOLIC BLOOD PRESSURE -W4S13Q3 LAST MO FELT THINGS GO YOUR WAY-W5S26Q5 12 MO,OFT SELL DRUGS-W3S14Q2 HOW OFTEN FEEL ISOLATED-W4S10Q0A PAST 7 DAY SHAKE BLUE-W5S28Q49 SINCE JUN95 HAVE DRIVEN DRUNK-W3S10Q0B PAST 7 DAY DEPRESSED-W5S22Q17 EVER INCARCERATED-W4
overall_rank.bootstrapoverall_rank.finalPermutation Importance of Predictors: Final vs. BootstrapPlacementsPredictorsmodel

Now, we create the final model for LASSO. There is no substantial difference between this method and the bootstrap methods, other than the data upon which the model is being built.


final_lasso_perf = NULL
  
# read file array
final_lasso_perfs = load_from_csv(final_lasso_perf, results_directory, no_tasks)

# get the index of the best performance and best performance
final_lassomodel_ind <- which.max(dplyr::select(final_lasso_perfs, met) %>% pull(met))
final_lasso_perf <- final_lasso_perfs %>% slice(final_lassomodel_ind)

# load the model of interest
final_model_lasso <- load_best_model('final_lasso_model', results_directory, final_lassomodel_ind)

The final LASSO performance metrics are shown below:

# show model final performance
print(final_lasso_perf)
ABCDEFGHIJ0123456789
accuracy
<dbl>
kap
<dbl>
sens
<dbl>
spec
<dbl>
ppv
<dbl>
0.93729730.19472250.2656250.96136620.1976744
1 row | 1-5 of 26 columns

Features: permutation importance


final_lasso_perm_plt = NULL

#load best index permutation from file  
final_lasso_perm_plt = load_from_csv(final_lasso_perm_plt, results_directory, best_ind=final_lassomodel_ind)

final_lasso_perm <- final_lasso_perm_plt %>%
  get_permute_placement(metric_oi=met) %>%
  add_attribute_names('predictor', full_dataset) %>%
  dplyr::select(predictor, everything())

head(final_lasso_perm, 20)
ABCDEFGHIJ0123456789
predictor
<chr>
att_name
<chr>
overall_rank
<int>
pr_auc
<dbl>
h5cj1cS14Q1C 12 MO SELL DRUGS-W510.055758799
h3to49S28Q49 SINCE JUN95 HAVE DRIVEN DRUNK-W320.016530114
h4mh2S14Q2 HOW OFTEN FEEL ISOLATED-W430.012293699
h1ds3S29Q3 LIE TO PARENTS ABOUT WHEREABOUT-W140.012094270
h4ds5S21Q5 12 MO,OFT SELL DRUGS-W450.011846507
age_w5age_w560.010822169
h4ds6S21Q6 12 MO,OFT STEAL SOMETHING/<$50-W470.010262974
hdlhdl80.006980847
h4mh18S14Q18 PAST 7 DAYS BOTHERED BY THINGS-W490.006813132
h4mh6S14Q6 LST MNTH DIFFICULTIES OVERWHELM-W4100.006293066
Next
12
Previous
1-10 of 20 rows
plot_permute_var_imp(final_lasso_perm, metric = met)

Comparison with bootstrap results

This section investigates the differences in the bootstrap results vs the features generated from the LASSO final model. The following table shows the overall differences in rank.

lasso_joined_results <- final_lasso_perm %>%
  dplyr::select(-met) %>%
  full_join(dplyr::select(bs_lasso_perm, -met), by=c("predictor", "att_name"), suffix=c('.final', '.bootstrap')) %>%
  mutate(mean_rank = (overall_rank.final + overall_rank.bootstrap)/2) %>%
  arrange(mean_rank)

head(lasso_joined_results, 20)
ABCDEFGHIJ0123456789
predictor
<chr>
att_name
<chr>
overall_rank.final
<int>
overall_rank.bootstrap
<int>
mean_rank
<dbl>
h5cj1cS14Q1C 12 MO SELL DRUGS-W5111.0
h4mh2S14Q2 HOW OFTEN FEEL ISOLATED-W4333.0
h4ds5S21Q5 12 MO,OFT SELL DRUGS-W4523.5
h3to49S28Q49 SINCE JUN95 HAVE DRIVEN DRUNK-W3295.5
h4ds6S21Q6 12 MO,OFT STEAL SOMETHING/<$50-W4756.0
h1ds3S29Q3 LIE TO PARENTS ABOUT WHEREABOUT-W14128.0
hdlhdl8109.0
h5mn3S13Q3 LAST MO FELT THINGS GO YOUR WAY-W514811.0
h4mh18S14Q18 PAST 7 DAYS BOTHERED BY THINGS-W491512.0
age_w5age_w562013.0
Next
12
Previous
1-10 of 20 rows

The following plot provides visualizations for the difference in the final model rankings vs the bootstrap.

# Comparison of top_n features
lasso_joined_results %>%
  compare_feature_select(sel_cols = c("overall_rank.final", "overall_rank.bootstrap"),
    interactive = TRUE,
    top_n = 100,
    opacity = 0.50,
    plot_title = "Permutation Importance of Predictors: Final vs. Bootstrap")
0306090S21Q4 12 MO,OFT WEAPON GET SOMETHING-W4S26Q18A 12 MO,SAW ONE SHOOT/STAB PERS-W3S26Q18I 12 MO,SHOT/STABBED SOMEONE-W3S30Q9 DRIVE WHILE DRUNK-W1S10Q0C PAST 7 DAY HAPPY-W5S14Q19 PAST 7 DAYS SHAKE OFF BLUES-W4S21Q14 12 MO,SAW ONE SHOOT/STAB PERS-W4S31Q8 SHOT/STABBED SOMEONE-W1S26Q18H 12 MO,PULL KNIKE/GUN SOMEONE-W3S27 MEASURED WAIST (CM) -W4S14Q22 PAST 7 DAYS FELT DEPRESSED-W4S26Q4 12 MO,OFT WEAPON GET SOMETHING-W3S10Q4 JUST AS GOOD AS OTHER PEOPLE-W1S10Q6 FELT DEPRESSED-W1S14Q4 LST MNTH CONF HANDLE PERS PROB-W4S26Q6 12 MO,OFT STEAL SOMETHING/<$50-W3S14Q25 PAST 7 DAYS ENJOYED LIFE-W4S14Q26 PAST 7 DAYS FELT SAD-W4S12Q13 PAST 7 DAYS PEOPLE DISLIKED U-W3S21Q20 12 MO,SHOT/STABBED SOMEONE-W4S5Q9 EXPELLED FROM SCHOOL-W1S14Q27 PAST 7 DAYS FELT DISLIKED-W4hba1cS31Q7 PULLED A KNIFE/GUN ON SOMEONE-W1S10Q19 LIFE NOT WORTH LIVING-W1S38Q14 CHANCES-KILLED BY AGE 21-W1S5Q7 RECEIVED OUT-OF-SCHL SUSPENSION-W1S31Q1 SAW SHOOTING/STABBING OF PERSON-W1S06Q5J EVER BEEN DX PANIC/ANXIETY DIS-W4S26Q30 DO NOT WORRY ABOUT PAST-W4S29Q11 USE OR THREATEN WITH A WEAPON-W1S9Q3 EXPECT MORE GOOD-W5S10Q0A PAST 7 DAY SHAKE BLUE-W5S12Q10 PAST 7 DAYS TOO TIRED DO THNGS-W3S21Q7 12 MO,OFT PART PHYS FIGHT/GRP-W4S27 BLOOD PRESSURE CLASS -W4S12Q9 PAST 7 DAYS WERE DEPRESSED-W3S12Q5 PAST 7 DAYS BOTHERED BY THINGS-W3S22Q17 EVER INCARCERATED-W4BIOLOGICAL SEX-W1S10Q1 BOTHERED BY THINGS-W1S9Q2 NOT EXPECT THINGS MY WAY-W5S10Q11 HAPPY-W1S38Q12 CHANCES-LIVE TO AGE 35-W1S10Q0D PAST 7 DAY SAD-W5S29Q14 TAKE PART IN A GROUP FIGHT-W1S29Q12 SELL DRUGS-W1S10Q15 ENJOYED LIFE-W1S10Q3 HAD THE BLUES-W1S26Q15 DO NOT EXPECT THINGS MY WAY-W4S10Q7 TOO TIRED TO DO THINGS-W1S14Q5 LST MNTH THINGS GOING YOUR WAY-W4S29Q13 STEAL WORTH < $50-W1crpS26Q23 EXPECT MORE GOOD THAN BAD-W4S14Q3 LST MNTH UNABLE CONTROL THINGS-W4S10Q16 FELT SAD-W1S12Q1 PAST 7 DAYS SHAKE OFF BLUES-W3S21Q19 12 MO,PULL KNIFE/GUN SOMEONE-W4S12Q11 PAST 7 DAYS ENJOYED LIFE-W3S12Q8 PAST 7 DAYS TROUBLE CONCENTRTNG-W3S27 SYSTOLIC BLOOD PRESSURE-W4S26Q6 WORRY ABOUT THINGS-W4S13Q4 LAST MO DIFFS OVERWHELM-W5S5Q2 FREQ-SKIPPED SCHOOL-W1S14Q1B 12 MO STEAL SOMETHING/>$50-W5S26Q31 RARELY EXPECT GOOD HAPPN TO ME-W4S12Q12 PAST 7 DAYS WERE SAD-W3S26Q22 I GET STRESSED OUT EASILY-W4S13Q1 LAST MO NO CNTRL IMPORT THINGS-W5S10Q0E PAST 7 DAY NOT WORTH-W5S13Q2 LAST MO CONFID HANDLE PERS PBMS-W5S12Q7 PAST 7 DAYS FELT AS GOOD AS OTH-W3ldltgS14Q1D 12 MO GET IN PHYS FIGHT-W5S10Q0B PAST 7 DAY DEPRESSED-W5S10Q5 TROUBLE KEEPING MIND FOCUSED-W1S27 DIASTOLIC BLOOD PRESSURE -W4one_race5S14Q20 PAST 7 DAYS FELT AS GOOD OTHRS-W4S10Q17 FELT PEOPLE DISLIKE YOU-W1S9Q1 ALWAYS OPTIM ABOUT FUT-W5S26Q7 ALWAYS OPTIMISTIC ABOUT FUTURE-W4S27 BMI -W4S14Q23 PAST 7 DAYS FELT TOO TIRED-W4S26Q5 12 MO,OFT SELL DRUGS-W3S14Q24 PAST 7 DAYS FELT HAPPY-W4S14Q6 LST MNTH DIFFICULTIES OVERWHELM-W4age_w5S26Q14 NOT EASILY BOTHERED BY THINGS-W4S14Q18 PAST 7 DAYS BOTHERED BY THINGS-W4S13Q3 LAST MO FELT THINGS GO YOUR WAY-W5hdlS29Q3 LIE TO PARENTS ABOUT WHEREABOUT-W1S21Q6 12 MO,OFT STEAL SOMETHING/<$50-W4S28Q49 SINCE JUN95 HAVE DRIVEN DRUNK-W3S21Q5 12 MO,OFT SELL DRUGS-W4S14Q2 HOW OFTEN FEEL ISOLATED-W4S14Q1C 12 MO SELL DRUGS-W5
overall_rank.bootstrapoverall_rank.finalPermutation Importance of Predictors: Final vs. BootstrapPlacementsPredictorsmodel

Here, we compare the features generated by the permutation importance between the two final models.

rf_lasso_final_joined_results <- final_rf_perm %>%
  dplyr::select(-met) %>%
  full_join(dplyr::select(final_lasso_perm, -met), by=c("predictor", "att_name"), suffix=c('.rf', '.lasso')) %>%
  mutate(mean_rank = (overall_rank.rf+overall_rank.lasso)/2) %>%
  arrange(mean_rank)

head(rf_lasso_final_joined_results, 20)
ABCDEFGHIJ0123456789
predictor
<chr>
att_name
<chr>
overall_rank.rf
<int>
overall_rank.lasso
<int>
mean_rank
<dbl>
h3to49S28Q49 SINCE JUN95 HAVE DRIVEN DRUNK-W3423.0
h4mh2S14Q2 HOW OFTEN FEEL ISOLATED-W4534.0
h5mn3S13Q3 LAST MO FELT THINGS GO YOUR WAY-W571410.5
age_w5age_w515610.5
h1ds3S29Q3 LIE TO PARENTS ABOUT WHEREABOUT-W124414.0
h4ds6S21Q6 12 MO,OFT STEAL SOMETHING/<$50-W422714.5
h4dbpS27 DIASTOLIC BLOOD PRESSURE -W4102015.0
h3ds5S26Q5 12 MO,OFT SELL DRUGS-W3181516.5
h5ss0bS10Q0B PAST 7 DAY DEPRESSED-W523317.5
h4mh18S14Q18 PAST 7 DAYS BOTHERED BY THINGS-W433921.0
Next
12
Previous
1-10 of 20 rows

The following visualization provides the intuition about the differences in the rankings between the final model types. They’re ordered by the overall mean importance, and for a given variable, the differences in rank are shown.

# Comparison of top_n features
rf_lasso_final_joined_results %>%
  compare_feature_select(sel_cols = c("overall_rank.rf", "overall_rank.lasso"),
    interactive = TRUE,
    top_n = 100,
    opacity = 0.50,
    plot_title = "Permutation Importance of Predictors: Random Forest vs Lasso")
0306090S06Q5J EVER BEEN DX PANIC/ANXIETY DIS-W4S15Q56 CHANCE OF-LIVING TILL AGE 35-W3S31Q7 PULLED A KNIFE/GUN ON SOMEONE-W1S06Q5H EVER BEEN DX WITH DEPRESSION-W4S31Q8 SHOT/STABBED SOMEONE-W1S14Q19 PAST 7 DAYS SHAKE OFF BLUES-W4S14Q22 PAST 7 DAYS FELT DEPRESSED-W4S21Q14 12 MO,SAW ONE SHOOT/STAB PERS-W4S21Q7 12 MO,OFT PART PHYS FIGHT/GRP-W4S26Q18A 12 MO,SAW ONE SHOOT/STAB PERS-W3S12Q13 PAST 7 DAYS PEOPLE DISLIKED U-W3S5Q9 EXPELLED FROM SCHOOL-W1S38Q12 CHANCES-LIVE TO AGE 35-W1S14Q26 PAST 7 DAYS FELT SAD-W4S31Q1 SAW SHOOTING/STABBING OF PERSON-W1h3bmiS10Q19 LIFE NOT WORTH LIVING-W1S29Q11 USE OR THREATEN WITH A WEAPON-W1S9Q3 EXPECT MORE GOOD-W5S14Q1B 12 MO STEAL SOMETHING/>$50-W5S9Q2 NOT EXPECT THINGS MY WAY-W5S10Q3 HAD THE BLUES-W1S14Q25 PAST 7 DAYS ENJOYED LIFE-W4S26Q6 12 MO,OFT STEAL SOMETHING/<$50-W3S12Q5 PAST 7 DAYS BOTHERED BY THINGS-W3S5Q6I EVER BEEN DX PANIC/ANXIETY DIS-W5S10Q4 JUST AS GOOD AS OTHER PEOPLE-W1S10Q6 FELT DEPRESSED-W1S14Q21 PAST 7 DAYS TROUBL CONCENTRTNG-W4S26Q30 DO NOT WORRY ABOUT PAST-W4S29Q13 STEAL WORTH < $50-W1S29Q14 TAKE PART IN A GROUP FIGHT-W1S10Q11 HAPPY-W1S27 BLOOD PRESSURE CLASS -W4S26Q23 EXPECT MORE GOOD THAN BAD-W4S12Q10 PAST 7 DAYS TOO TIRED DO THNGS-W3crpS10Q16 FELT SAD-W1S14Q4 LST MNTH CONF HANDLE PERS PROB-W4S38Q14 CHANCES-KILLED BY AGE 21-W1S5Q6G EVER BEEN DX WITH DEPRESSION-W5S5Q2 FREQ-SKIPPED SCHOOL-W1S21Q19 12 MO,PULL KNIFE/GUN SOMEONE-W4S21Q5 12 MO,OFT SELL DRUGS-W4S26Q15 DO NOT EXPECT THINGS MY WAY-W4S29Q12 SELL DRUGS-W1BIOLOGICAL SEX-W1S14Q3 LST MNTH UNABLE CONTROL THINGS-W4S12Q1 PAST 7 DAYS SHAKE OFF BLUES-W3S26Q22 I GET STRESSED OUT EASILY-W4S5Q7 RECEIVED OUT-OF-SCHL SUSPENSION-W1S10Q0A PAST 7 DAY SHAKE BLUE-W5S22Q17 EVER INCARCERATED-W4S10Q15 ENJOYED LIFE-W1S10Q1 BOTHERED BY THINGS-W1S27 MEASURED WAIST (CM) -W4S10Q0C PAST 7 DAY HAPPY-W5S14Q5 LST MNTH THINGS GOING YOUR WAY-W4S26Q6 WORRY ABOUT THINGS-W4S12Q9 PAST 7 DAYS WERE DEPRESSED-W3S10Q17 FELT PEOPLE DISLIKE YOU-W1ldlS14Q1D 12 MO GET IN PHYS FIGHT-W5S12Q12 PAST 7 DAYS WERE SAD-W3S14Q1C 12 MO SELL DRUGS-W5S30Q9 DRIVE WHILE DRUNK-W1S12Q7 PAST 7 DAYS FELT AS GOOD AS OTH-W3S13Q1 LAST MO NO CNTRL IMPORT THINGS-W5S26Q14 NOT EASILY BOTHERED BY THINGS-W4S13Q4 LAST MO DIFFS OVERWHELM-W5S10Q0D PAST 7 DAY SAD-W5one_race5tgS12Q8 PAST 7 DAYS TROUBLE CONCENTRTNG-W3S10Q7 TOO TIRED TO DO THINGS-W1S26Q31 RARELY EXPECT GOOD HAPPN TO ME-W4S26Q7 ALWAYS OPTIMISTIC ABOUT FUTURE-W4hdlS9Q1 ALWAYS OPTIM ABOUT FUT-W5S12Q11 PAST 7 DAYS ENJOYED LIFE-W3S14Q24 PAST 7 DAYS FELT HAPPY-W4S27 BMI -W4S10Q5 TROUBLE KEEPING MIND FOCUSED-W1S14Q20 PAST 7 DAYS FELT AS GOOD OTHRS-W4S14Q23 PAST 7 DAYS FELT TOO TIRED-W4S14Q27 PAST 7 DAYS FELT DISLIKED-W4S10Q0E PAST 7 DAY NOT WORTH-W5S27 SYSTOLIC BLOOD PRESSURE-W4S14Q6 LST MNTH DIFFICULTIES OVERWHELM-W4S13Q2 LAST MO CONFID HANDLE PERS PBMS-W5S14Q18 PAST 7 DAYS BOTHERED BY THINGS-W4S10Q0B PAST 7 DAY DEPRESSED-W5S26Q5 12 MO,OFT SELL DRUGS-W3S27 DIASTOLIC BLOOD PRESSURE -W4S21Q6 12 MO,OFT STEAL SOMETHING/<$50-W4S29Q3 LIE TO PARENTS ABOUT WHEREABOUT-W1age_w5S13Q3 LAST MO FELT THINGS GO YOUR WAY-W5S14Q2 HOW OFTEN FEEL ISOLATED-W4S28Q49 SINCE JUN95 HAVE DRIVEN DRUNK-W3
overall_rank.lassooverall_rank.rfPermutation Importance of Predictors: Random Forest vs LassoPlacementsPredictorsmodel

With the final models generated, we’re now able to compare their performance metrics.

# Comparison of performance metrics
valid_perf <- get_metric_set_from_perfs(perf_list = list(final_rf_perf, final_lasso_perf)) %>%
  mutate(model = c('rf', 'lasso'))

testing_perf <- get_metric_set_from_models(testing_df, list(final_model_rf, final_model_lasso), out=outcome) %>%
  mutate(model = c('rf', 'lasso'))

Validation and selection. The following table shows the comparison between models in terms of the validation set. We can select our final model based on the best performing model according to the metric.

print(valid_perf)
ABCDEFGHIJ0123456789
model
<chr>
accuracy
<dbl>
kap
<dbl>
sens
<dbl>
spec
<dbl>
rf0.93729730.27862640.4062500.9563270
lasso0.93729730.19472250.2656250.9613662
2 rows | 1-5 of 27 columns

Testing performance. The following shows the performance of both the models on the test set. Note that although we don’t use this test set to evaluate the final models, we can still see how our selected method would have performed.

print(testing_perf)
ABCDEFGHIJ0123456789
model
<chr>
accuracy
<dbl>
kap
<dbl>
sens
<dbl>
spec
<dbl>
rf0.94162160.22023790.29032260.9642058
lasso0.94810810.14575960.16129030.9753915
2 rows | 1-5 of 27 columns

The following plots show a comparison between the performance of the models on the validation and test sets. Again, we don’t choose the model based on the test set, but curiosity dictates that we view this performance.

# Show plots side by side
metrics_of_interest = c('model', 'accuracy', 'bal_accuracy', 'mpce', 'sens', 'spec', 'ppv', 'npv', 'pr_auc', 'roc_auc')
valid_plt <- plot_metric_set(dplyr::select(valid_perf, all_of(metrics_of_interest)), plot_title = "Model comparison for validation set")
test_plt <- plot_metric_set(dplyr::select(testing_perf, all_of(metrics_of_interest)), plot_title = "Model comparison for testing set")
gridExtra::grid.arrange(gridExtra::arrangeGrob(valid_plt, test_plt, ncol=2, nrow=1))

Other cleanup

h2o.shutdown(FALSE)
LS0tCnRpdGxlOiAiODQtaW50ZXJwcmV0LWlsbC1kcnVnIgpvdXRwdXQ6CiAgaHRtbF9ub3RlYm9vazoKICAgIGNvZGVfZm9sZGluZzogaGlkZQogICAgdGhlbWU6IGx1bWVuCiAgICB0b2M6IHllcwogICAgdG9jX2Zsb2F0OiB5ZXMKICAgIHRvY19kZXB0aDogNAogIGh0bWxfZG9jdW1lbnQ6CiAgICBkZl9wcmludDogcGFnZWQKICAgIHRvYzogeWVzCmVkaXRvcl9vcHRpb25zOgogIGNodW5rX291dHB1dF90eXBlOiBpbmxpbmUKLS0tCgpgYGB7ciBzZXR1cCwgaW5jbHVkZT1GQUxTRX0Ka25pdHI6Om9wdHNfY2h1bmskc2V0KGVjaG8gPSBUUlVFLCB3YXJuaW5nPUZBTFNFLCBtZXNzYWdlPUZBTFNFKQpgYGAKCmBgYHtyIHNvdXJjZSBmaWxlcywgaW5jbHVkZT1GQUxTRX0Kc291cmNlKCJmdW5jdGlvbl9pbXBvcnQuUiIpCmBgYAoKIyMgUHJvamVjdCBydW4gcGFyYW1ldGVycwpgYGB7ciBzZWVkcyBmb3IgcmVwcm9kdWNpYmlsaXR5fQpub190YXNrcz0yCnNlZWQgPSA5Mzg0Cm91dGNvbWUgPSAnaV9kcnVnJwpmYmFzZSA9ICcvc2NyYXRjaC9wX2dheWRvc2hfbGFiL0RTSS8nCnJlc3VsdHNfZGlyZWN0b3J5IDwtIHN0cl9jKGZiYXNlLCBvdXRjb21lLCAnL3Jlc3VsdHNfZnVsbF9oMm9fMl9kZWxldGVkJykKc2V0LnNlZWQoc2VlZCkKYGBgCgoqKlB1cnBvc2UuKiogSW4gdGhpcyB3b3JrLCB3ZSB3aWxsIGV4cGxvcmUgdGhlIHJlbGF0aW9uIGJldHdlZW4gaWRlbnRpZmllZCBtZWFzdXJlcyBvZiBkZXNwYWlyIG9mIGludGVyZXN0IChlLmcuLCBwZXJzb25hbGl0eSBtZWFzdXJlcyBvZiBzZWxmLWNvbnNjaW91c25lc3MsIGluZGl2aWR1YWwgYW5kIGNvbXBvc2l0ZSBpdGVtIHNjb3JlcyBmcm9tIHRoZSBDRVMtRCBhc3Nlc3NtZW50KSBhbmQgZGVzY3JpcHRvcnMgb2YgZGlzZWFzZXMgb2YgZGVzcGFpci4gIFdlIHdpbGwgYWNoaWV2ZSB0aGlzIGdvYWwgdGhyb3VnaCBtb2RlbGluZyB0aGUgb3V0Y29tZXMgYmFzZWQgb24gdGhlIGluY2x1ZGVkIHByZWRpY3RvcnMsIGFuZCByb2J1c3RseSBhc3Nlc3MgdGhlIGltcG9ydGFuY2Ugb2YgdGhlIGluY2x1ZGVkIGZlYXR1cmVzIGluIHByZWRpY3RpbmcgdGhlIG91dGNvbWVzIHZpYSBib290c3RyYXBwaW5nLiAgV2Ugd2lsbCB1c2UgdHdvIHdlbGwta25vd24gbWFjaGluZSBsZWFybmluZyBtb2RlbHMsIHJhbmRvbSBmb3Jlc3RzIGFuZCBMQVNTTywgd2hpY2ggYXJlIGJvdGggZnJlcXVlbnRseSB1c2VkIHRvIG1lYXN1cmUgdGhlIHJlbGF0aXZlIGltcG9ydGFuY2Ugb2YgdGhlIHByZWRpY3RvcnMgaW5jbHVkZWQgaW4gdGhlIG1vZGVscy4gIExhc3RseSwgd2UnbGwgZ2VuZXJhdGUgdHJhaW5lZCBhbmQgdHVuZWQgbW9kZWxzIHVzaW5nIHRoaXMgcmVkdWNlZCBmZWF0dXJlIHNldCB3aGljaCBjYW4gYmUgdXNlZCBieSBvdGhlcnMgd2lzaCB0byBwcmVkaWN0IHRoZSBpZGVudGlmaWVkIG91dGNvbWVzLgoKKipTdWJqZWN0IGluY2x1c2lvbi4qKiBGb3IgdGhpcyBpbnZlc3RpZ2F0aW9uLCB3ZSB3aWxsIG9taXQgdGhlIGVudGlyZXR5IG9mIFdhdmUgMi4gIFRoaXMgaXMgY29tbW9ubHkgZG9uZSBpbiBhbmFseXNlcyBvZiBBZGRIZWFsdGggZGF0YSBkdWUgdGhlIGRlc2lnbiBvZiB0aGUgb3JpZ2luYWwgc3R1ZHkuICBPdGhlcndpc2UsIG91ciBkYXRhc2V0IHdpbGwgaW5jbHVkZSBvbmx5IHN1YmplY3RzIHdobyBoYXZlIHByZWRpY3RvciBhbmQgb3V0Y29tZSBkYXRhIGluIF9hbGxfIG9mIHRoZSB3YXZlcy4KCioqT3V0Y29tZSB2YXJpYWJsZXMuKiogSW4gdGhpcyBleHBlcmltZW50LCB3ZSBhc3Nlc3MgX3N1aWNpZGFsIGlkZWF0aW9uXyBhdCBXYXZlIDUuICAKCioqUHJlZGljdG9yIHZhcmlhYmxlcy4qKiBUaGUgcHJlZGljdG9ycyBmb3IgdGhlc2UgbW9kZWxzIGFyZSBoYW5kLXBpY2tlZCwgYW5kIGJhc2VkIG9uIHByZXZpb3VzIHdvcmssIHJlbGV2YW5jZSwgYW5kIHN1YmplY3QgbWF0dGVyIGV4cGVydGlzZS4gVGhlIHNldCBvZiBwcmVkaWN0b3JzIGFuZCB0aGUgc2V0IG9mIG91dGNvbWVzIGFyZSBkaXNqb2ludC4gIFByZWRpY3RvcnMgZnJvbSBXYXZlcyAxLTQgKGV4Y2x1ZGluZyBXYXZlIDIsIHNlZSBhYm92ZSkgYXJlIGluY2x1ZGVkLCBhbmQgd2lsbCBiZSBkZXRhaWxlZCBpbiB0aGUgZm9sbG93aW5nIGFuYWx5c2lzLgoKYGBge3IgbG9hZCBsaWJyYXJpZXMsIGluY2x1ZGU9RkFMU0V9CiMgVXNlIHBhY21hbiwgd2hpY2ggZm9yY2VzIGFuIGluc3RhbGwgaWYgdGhlIGxpYnJhcnkgaXNuJ3QgcHJlc2VudCBvbiB0aGUgcnVubmluZyBtYWNoaW5lCmlmICghcmVxdWlyZSgicGFjbWFuIikpIGluc3RhbGwucGFja2FnZXMoInBhY21hbiIpCiNwYWNtYW46OnBfaW5zdGFsbChwbG90bHkpCnBhY21hbjo6cF9sb2FkKHRpZHl2ZXJzZSwgaDJvLCBmdXJycikKCmBgYAoKYGBge3IgaW5pdGlhbGl6YXRpb25zLCBpbmNsdWRlPUZBTFNFfQpwb3J0X25vIDwtIHN0YXJ0X2gybygpCmgyby5ub19wcm9ncmVzcygpCmZ1dHVyZTo6cGxhbihtdWx0aXByb2Nlc3MpCmBgYAoKIyBQaXBlbGluZSBvdmVydmlldwpgYGB7ciBlY2hvPUZBTFNFLCBvdXQud2lkdGg9JzUwJSd9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKCcuL2ltZy9waXBlbGluZV9vdmVydmlldy5qcGcnKQpgYGAKCiMgRGF0YXNldCBnZW5lcmF0aW9uCgpUaGUgcHJlZGljdG9ycyB3ZSB3aWxsIGJlIHVzaW5nIHdpbGwgYmUgdGhlIHRoZSB2YXJpYWJsZSBgcHJlZGljdG9yX2xpc3RgIGxvYWRlZCBmcm9tIGAxMC1pbXBvcnQtZGF0YS5SbWRgIGZpbGUuIFRoZXNlIGluaXRpYWwgc2V0IG9mIHByZWRpY3RvcnMgd2lsbCBiZSBiYXNlZCBvZiB0aGUgbGlzdCBvZiB2YXJpYWJsZXMgdGhhdCBkZXNjcmliZSBhbnhpZXR5LCBkZXByZXNzaW9uLCBhbmQgb3B0aW1pc20uCgpgYGB7ciBsb2FkIHJhdyBkYXRhIGFuZCBmb3JtdWxhdGUgZGF0YXNldCwgd2FybmluZz1UUlVFLCBtZXNzYWdlPVRSVUV9CiMjIHNldCBvdXRjb21lIHZhcmlhYmxlIG9mIGludGVyZXN0CmZpbGViYXNlID0gJy9zY3JhdGNoL3BfZ2F5ZG9zaF9sYWInCgojY3JlYXRlIGRhdGEgaW4gc3BlY2lmaWVkIGZvcm0KZGF0YXNldF9saXN0IDwtIGdlbmVyYXRlX2RhdGFzZXRzKG91dGNvbWUsIGJpbmFyaXplPUZBTFNFLCBmaWxlYmFzZT1maWxlYmFzZSwgc2VlZF92YWw9c2VlZCkKCiNwYXJzZSBvdXQgZGF0YXNldCBjb21wb25lbnRzCndhdmVfZGF0YSA8LSBkYXRhc2V0X2xpc3Qkd2F2ZV9kYXRhCmZ1bGxfZGF0YXNldCA8LSBkYXRhc2V0X2xpc3QkZnVsbF9kYXRhc2V0CmRzX3JhdyA8LSBkYXRhc2V0X2xpc3QkZHNfcmF3X291dGNvbWUKZHMgPC0gZGF0YXNldF9saXN0JGRzX2ZpbmFsCgojbWwgc3BsaXRzIG9mIHRoZSBkYXRhCnRyYWluaW5nX2RmIDwtIGRhdGFzZXRfbGlzdCR0cmFpbmluZ19kZgp2YWxpZGF0aW9uX2RmIDwtIGRhdGFzZXRfbGlzdCR2YWxpZGF0aW9uX2RmCnRlc3RpbmdfZGYgPC0gZGF0YXNldF9saXN0JHRlc3RpbmdfZGYKCmBgYAoKIyMgT3V0Y29tZSB2YXJpYWJsZTogQmluYXJpemluZyBhbmQgcmVjb2RpbmcgZGV0YWlscwoKVGhlIGZvbGxvd2luZyB0YWJsZSBkZXRhaWxzIHRoZSB2YWx1ZXMgcHJlc2VudCBpbiB0aGUgb3V0Y29tZSB2YXJpYWJsZS4gIEFzIHdlIGNhbiBzZWUsIHdlIG5lZWQgdG8gY29udmVydCB0aGUgdmFyaWFibGUgaW50byBhIGZhY3RvciwgYW5kIGRyb3AgdGhlIE5Bcy4KCmBgYHtyIHdhcm5pbmc9RkFMU0UsIG1lc3NhZ2U9RkFMU0V9CmRzX3JhdyAlPiUgCiAgZ3JvdXBfYnkoLmRhdGFbW291dGNvbWVdXSkgJT4lIAogIHN1bW1hcmlzZSh0b3RhbCA9IG4oKSwgdHlwZSA9IGNsYXNzKC5kYXRhW1tvdXRjb21lXV0pKQoKZHMgJT4lIAogIGdyb3VwX2J5KC5kYXRhW1tvdXRjb21lXV0pICU+JSAKICBzdW1tYXJpc2UodG90YWwgPSBuKCksIHR5cGUgPSBjbGFzcyguZGF0YVtbb3V0Y29tZV1dKSkKCnByaW50KHN0cl9jKCdUb3RhbCBudW1iZXIgb2Ygc3ViamVjdHMgaW4gdW5wcm9jZXNzZWQgb3V0Y29tZSBkYXRhOiAnLCBucm93KGRzX3JhdykpKQpwcmludChzdHJfYygnVG90YWwgbnVtYmVyIG9mIHN1YmplY3RzIGluIHByb2Nlc3NlZCBvdXRjb21lIGRhdGE6ICcsIG5yb3coZHMpKSkKcHJpbnQoc3RyX2MoJ1RvdGFsIHN1YmplY3RzIGRyb3BwZWQgZHVlIHRvIE5BIG9yIHNraXA6ICcsIG5yb3coZHNfcmF3KS1ucm93KGRzKSkpCgpgYGAKClRoZSB0YWJsZSBhbHNvIGRlbW9uc3RyYXRlcyB0aGF0IHRoZSBjbGFzc2VzIGFyZSB2ZXJ5IGltYmFsYW5jZWQsIHdpdGggYWJvdXQgMTR4IHRoZSBuZWdhdGl2ZSBjbGFzcyBhcyBjb21wYXJlZCB3aXRoIHRoZSBwb3NpdGl2ZSBjbGFzcy4gIFRoaXMgaXMgZGlzcGxheWVkIGdyYXBoaWNhbGx5IGZvciBlYXN5IGNvbnN1bXB0aW9uIGJlbG93LgoKYGBge3IgZmlnLmhlaWdodD02LCBmaWcud2lkdGg9MTAsIHdhcm5pbmc9VFJVRX0KZHNfcmF3ICU+JQogIGV4cGxvcmVfb3V0Y29tZShkcywgb3V0Y29tZSkKCmBgYAoKQWZ0ZXIgZHJvcHBpbmcgdGhlIE5Bcywgd2Ugc2VlIHRoYXQgd2Ugbm93IGhhdmUgOTE2OCByb3dzLCB3aGljaCBpcyBjb25zaXN0ZW50IHdpdGggdGhlIHRhYmxlIGFib3ZlLiAgVGhlIGRpc3RyaWJ1dGlvbiBvZiB0aGUgZGF0YSBoYXMgYSBkaXN0aW5jdCBpbWJhbGFuY2UgYXMgbm90ZWQgYWJvdmUuICBJIHRoaW5rIHRoaXMgd2FycmFudHMgdXNpbmcgYHByX2F1Y2AgYXMgdGhlIG9wdGltaXphdGlvbiBhbmQgc2VsZWN0aW9uIG1ldHJpYy4KCiMgRGF0YSBleHBsb3JhdGlvbiBhbmQgdmlzdWFsaXphdGlvbgpIZXJlLCB3ZSBjb21tZW50IGFib3V0IHRoZSBnZW5lcmFsIGNoYXJhY3RlcmlzdGljcyBvZiB0aGUgZGF0YSBiYXNlZCBvbiB0aGUgcHJvdmlkZWQgdmlzdWFsaXphdGlvbnMuICBXZSBjb21tZW50IG9uIG1pc3NpbmduZXNzIG9mIGRhdGEsIGFueSBzdHJhbmdlIG9yIHVudXN1YWwgYmVoYXZpb3IgKGUuZy4sIHN0cm9uZyBpbWJhbGFuY2VzKSwgYW5kIGFueSBjb3JyZWxhdGlvbiB0aGF0IHN0aWNrcyBvdXQuCgpgYGB7ciB3YXJuaW5nPUZBTFNFLCBtZXNzYWdlPUZBTFNFfQojUmVwb3J0IGFib3V0IHRoZSBjaGFyYWN0ZXJpc3RpY3Mgb2YgdGhlIHN1YmplY3RzIGxlZnQgb3V0IG9mIHRoZSBqb2luCmRzICU+JSBleHBsb3JlX2Ryb3BwZWQoKQoKYGBgCgpgYGB7ciBlZGEgZGlzdHJpYnV0aW9ucywgZmlnLndpZHRoID0gMTUsIGZpZy5oZWlnaHQ9MTV9CiMgVmlzdWFsaXplIGRpc3RyaWJ1dGlvbnMgb2YgdmFyaWFibGVzIG9mIGludGVyZXN0CmRzICU+JSAKICBkcGx5cjo6c2VsZWN0KC1haWQpICU+JQogIGdyYXBoX2Jhcl9kaXNjcmV0ZShkZiA9IC4sCiAgICAgICAgICAgICAgICAgICAgIHBsb3RfdGl0bGUgPSAiRGlzdHJpYnV0aW9ucyBvZiBEaXNjcmV0ZSBWYXJpYWJsZXMiLAogICAgICAgICAgICAgICAgICAgICBtYXhfY2F0ZWdvcmllcyA9IDUwLAogICAgICAgICAgICAgICAgICAgICBudW1fcm93cyA9IDMsCiAgICAgICAgICAgICAgICAgICAgIG51bV9jb2xzID0gMywKICAgICAgICAgICAgICAgICAgICAgeF9heGlzX3NpemUgPSAxMiwKICAgICAgICAgICAgICAgICAgICAgeV9heGlzX3NpemUgPSAxMiwKICAgICAgICAgICAgICAgICAgICAgdGl0bGVfc2l6ZSA9IDE1KQpgYGAKCmBgYHtyIGVkYSBtaXNzaW5nfQpkcyAlPiUKICBncmFwaF9taXNzaW5nKG9ubHlfbWlzc2luZyA9IFRSVUUsCiAgICAgICAgICAgICAgICB0aXRsZSA9ICJQZXJjZW50IE1pc3NpbmciLAogICAgICAgICAgICAgICAgYm94X2xpbmVfc2l6ZSA9IC41LAogICAgICAgICAgICAgICAgbGFiZWxfc2l6ZSA9IC41LAogICAgICAgICAgICAgICAgeF9heGlzX3NpemUgPSAxMiwKICAgICAgICAgICAgICAgIHlfYXhpc19zaXplID0gMTIsCiAgICAgICAgICAgICAgICB0aXRsZV9zaXplID0gMTUpCgpgYGAKCmBgYHtyIGVkYSBjb3JyZWxhdGlvbnN9CmRzICU+JQogICNkcGx5cjo6c2VsZWN0KDE6MjApICU+JQogIHBhaXJ3aXNlX2NyYW1lcnNfdigpICU+JQogIHBsb3RfY3JhbWVyX3YoeF9heGlzX2FuZ2xlID0gOTAsCiAgICAgICAgICAgICAgICBwbG90X3RpdGxlID0gIkFzc29jaWF0aW9uIGFtb25nIENhdGVnb3JpY2FsIFZhcmlhYmxlcyIsCiAgICAgICAgICAgICAgICBpbnRlcmFjdGl2ZSA9IFRSVUUpCmBgYApUaGUgY29ycmVsYXRpb24gcGxvdCBzdWdnZXN0cyB0aGF0IHRoZXJlIGFyZSBzZXZlcmFsIHZhcmlhYmxlcyB3ZSBtaWdodCB0aGluayBhYm91dCByZW1vdmluZy4gIEZpcnN0bHksIHRoZXJlIGFyZSBtb2RlcmF0ZSBjb3JyZWxhdGlvbnMgYW1vbmcgdGhlIGluZGl2aWR1YWwgcHJlZGljdG9yIGJsb2NrczsgdGhpcyBpcyBkdWUgdG8gdGhlIHdheSB0aGF0IHRoZXkgYXJlIG9yZGVyZWQsIHNpbmNlIHRoZXkncmUgZXNzZW50aWFsbHkgZ3JvdXBlZCBpbnRvIHN1YnNldHMuICBBZGR0aW9uYWxseSwgc29tZSBwcmVkaWN0b3IgcGFpcnMgaGF2ZSBleHRyZW1lbHkgaGlnaCBjb3JyZWxhdGlvbnMsIGxpa2UgKGg0aWQ1aiwgaDRtaDI2KSAoY29ycmVsYXRpb24gb2YgMC43MiksIGFuZCBtYW55IHZhcmlhYmxlcyBpbiB0aGUgaDRtaCogc2VyaWVzLiAgKGgzaWQ1aiwgaDRpZDVoKSBhbHNvIGhhcyBoaWdoIGNvcnJlbGF0aW9uID4gMC41LiAgV2UgbWF5IHdhbnQgdG8gY29uc2lkZXIgcmVtb3Zpbmcgc2V2ZXJhbCBvZiB0aGVzZSBpbiBhZGRpdGlvbiB0byB0aGUgYWdlIHZhcmlhYmxlcyBiZWNhdXNlIGl0IG1heSBjYXVzZSBmZWF0dXJlIGltcG9ydGFuY2UgbWFza2luZyB3aXRoaW4gb3VyIGFwcHJvYWNoZXMuCgojIFJvYnVzdCBmZWF0dXJlIGV2YWx1YXRpb24gey50YWJzZXQgLnRhYnNldC1mYWRlIC50YWJzZXQtcGlsbHN9CgpUaGUgZm9sbG93aW5nIHRhYmxlIGRpc3BsYXlzIHRoZSBtZWFuIHBlcmZvcm1hbmNlIG1ldHJpY3MgZm9yIHRoZSBib290c3RyYXBwZWQgbW9kZWxzIG9uIHRoZSB2YWxpZGF0aW9uIHNldCwgcmVtb3ZpbmcgdmFsdWVzIGZvciB3aGljaCB0aGVyZSBhcmUgTkEuCgpgYGB7ciBzYXZlL2xvYWQgcmYgYm9vdHN0cmFwIG1vZGVsIHBlcmZvcm1hbmNlLCBpbmNsdWRlPUZBTFNFfQoKYnNfcmZfcGVyZiA9IE5VTEwKCiMgZ2V0IGRhdGEgZmlsZXMKYnNfcmZfcGVyZiA9IGxvYWRfZnJvbV9jc3YoYnNfcmZfcGVyZiwgcmVzdWx0c19kaXJlY3RvcnksIG5vX3Rhc2tzKQoKYGBgCgoKYGBge3IgZXZhbHVhdGUgYm9vdHN0cmFwIG1vZGVsIHBlcmZvcm1hbmNlIHJmLCB3YXJuaW5nPUZBTFNFLCBtZXNzYWdlPUZBTFNFfQoKbWVhbl9ic19yZl9wZXJmIDwtIGJzX3JmX3BlcmYgJT4lCiAgc3VtbWFyaXNlX2lmKGlzLm51bWVyaWMsIG1lYW4sIG5hLnJtPVRSVUUpICU+JQogIG11dGF0ZShtb2RlbCA9ICdic19yZicpICU+JQogIGRwbHlyOjpzZWxlY3QobW9kZWwsIGV2ZXJ5dGhpbmcoKSkKCm1lYW5fYnNfcmZfcGVyZgpgYGAKQXMgc2hvd24sIHRoZSBib290c3RyYXBwZWQgbW9kZWxzIHRlbmQgdG8gaGF2ZSBoaWdoIHNwZWNpZmljaXR5IGJ1dCBsb3cgc2Vuc2l0aXZpdHksIGluZGljYXRpbmcgdGhhdCB0aGVyZSBpcyBhIGNoYWxsZW5nZSBpbiBpZGVudGlmeWluZyBzdWJqZWN0cyB3aXRoIHN1aWNpZGFsIGlkZWF0aW9uLgoKIyMjIEZlYXR1cmUgaW1wb3J0YW5jZXM6IFJhbmRvbSBGb3Jlc3QKIyMjIyBNZWFuIGRlY3JlYXNlIGluIGltcHVyaXR5IChNREkpCgpgYGB7ciBzYXZlL2xvYWQgcmYgYm9vdHN0cmFwIG1vZGVsIG1kaSwgaW5jbHVkZT1GQUxTRX0KCmJzX3JmX21kaSA9IE5VTEwKICAKIyBnZXQgZGF0YSBmaWxlcwpic19yZl9tZGkgPSBsb2FkX2Zyb21fY3N2KGJzX3JmX21kaSwgcmVzdWx0c19kaXJlY3RvcnksIG5vX3Rhc2tzKQoKCmBgYAoKYGBge3IgZXZhbHVhdGUgYm9vdHN0cmFwIG1kaX0KYm9vdF9yZl9tZGkgPC0gbGlzdChic19yZl9tZGkpICU+JQogIGdldF9tZWRpYW5fcGxhY2VtZW50KHVzZV9iYXNlX3ZhciA9IFRSVUUpICU+JQogIGFkZF9hdHRyaWJ1dGVfbmFtZXMoJ3ByZWRpY3RvcicsIGZ1bGxfZGF0YXNldCkgJT4lCiAgZHBseXI6OnNlbGVjdChwcmVkaWN0b3IsIGF0dF9uYW1lLCBvdmVyYWxsX3JhbmspCgpoZWFkKGJvb3RfcmZfbWRpLCAyMCkKYGBgClRoaXMgdGFibGUgcmV0dXJucyB0aGUgTURJIHZhcmlhYmxlIGltcG9ydGFuY2UgcmFua3MgdGhhdCByZXR1cm5lZCBmcm9tIGVhY2ggb2YgdGhlIGJvb3RzdHJhcHBlZCBtb2RlbHMuCgpgYGB7ciBmaWcud2lkdGggPSAxMCwgZmlnLmhlaWdodCA9IDEyfQojIE5lZWRzIHRvIGJlIGZpeGVkIHNvIHRoYXQgYXhlcyBkb24ndCBvdmVybGFwIGVhY2ggb3RoZXIgYW5kIG9ic2N1cmUgdW5kZXJzdGFuZGluZwpwbG90X3BsYWNlbWVudF9ib3hwbG90KGxpc3QoYnNfcmZfbWRpKSkKYGBgCgojIyMjIFBlcm11dGF0aW9uIGltcG9ydGFuY2UKTm93LCBsZXQncyBsb29rIGF0IHRoZSBwZXJtdXRhdGlvbiBpbXBvcnRhbmNlOgoKYGBge3IgY29tcHV0ZS9zYXZlIG9yIGxvYWQgYnMgcmYgbW9kZWwgcGVybXV0YXRpb24sIGluY2x1ZGU9RkFMU0V9Cgpic19yZl9wZXJtX3BsdCA8LSBOVUxMCiAgCiMgZ2V0IGRhdGEgZmlsZXMKYnNfcmZfcGVybV9wbHQgPC0gbG9hZF9mcm9tX2Nzdihic19yZl9wZXJtX3BsdCwgcmVzdWx0c19kaXJlY3RvcnksIG5vX3Rhc2tzKQoKYGBgCgoKYGBge3IgYWdncmVnYXRlIHJmIHBlcm0gcmVzdWx0c30KbWV0IDwtICdwcl9hdWMnCmJzX3JmX3Blcm0gPC0gYnNfcmZfcGVybV9wbHQgJT4lCiAgZ2V0X3Blcm11dGVfcGxhY2VtZW50KG1ldHJpY19vaT1tZXQpICU+JQogIGFkZF9hdHRyaWJ1dGVfbmFtZXMoJ3ByZWRpY3RvcicsIGZ1bGxfZGF0YXNldCkgJT4lCiAgZHBseXI6OnNlbGVjdChwcmVkaWN0b3IsIGV2ZXJ5dGhpbmcoKSkKCmhlYWQoYnNfcmZfcGVybSwgMjApCmBgYAoKIyMjIyBNREkgdnMgUGVybXV0YXRpb24gaW1wb3J0YW5jZQpJbiB0aGlzIHN0ZXAsIHdlIGFzc2VzcyB0aGUgZGlmZmVyZW5jZXMgZ2VuZXJhdGVkIGJldHdlZW4gdGhlIGRpZmZlcmVudCB0eXBlcyBvZiBpbXBvcnRhbmNlcy4KYGBge3IgZmlnLndpZHRoID0gMTYsIGZpZy5oZWlnaHQgPSAxNH0KY2JpbmQoYm9vdF9yZl9tZGlbMToyMCxdLCBkcGx5cjo6c2VsZWN0KGJzX3JmX3Blcm1bMToyMCxdLCAtYWxsX29mKG1ldCkpKQpgYGAKQXMgc2hvd24sIHRoZSBNREkgaW1wb3J0YW5jZSBzdWZmZXJzIGZyb20gaW1iYWxhbmNlcyBkdWUgdG8gdGhlIG51bWJlciBvZiB2YWx1ZXMgYXNzb2NpYXRlZCB3aXRoIGEgcHJlZGljdG9yLiAgQmVjYXVzZSB0aGUgd2F2ZSBhZ2VzIGhhdmUgc28gbWFueSBtb3JlIHZhbHVlcyB0aGFuIHRoZSBvdGhlciBmYWN0b3JzLCB0aGlzIGFydGlmaWNpYWxseSBpbmZsYXRlcyB0aGVpciBpbXBvcnRhbmNlIGluIE1ESS4gIFRoZSBwZXJtdXRhdGlvbiBpbXBvcnRhbmNlIGlzIG1vcmUgaW50dWl0aXZlLgoKYGBge3IgZmlnLndpZHRoID0gMTAsIGZpZy5oZWlnaHQgPSAxMn0KcGxvdF9wZXJtdXRlX3Zhcl9pbXAoYnNfcmZfcGVybSwgbWV0cmljID0gbWV0KQpgYGAKCgojIyBMQVNTTyBtb2RlbApJbiB0aGlzIHN0ZXAsIHdlIG1vZGVsIHRoZSByZWxhdGlvbiBiZXR3ZWVuIHRoZSBvdXRjb21lcyBhbmQgdGhlIHByZWRpY3RvcnMgdXNpbmcgYSBsaW5lYXIgcmVncmVzc2lvbiB3aXRoIEwyIHJlZ3VsYXJpemF0aW9uLiAgVGhpcyBkcml2ZXMgdGhlIGltcG9ydGFuY2Ugb2YgdW5pbXBvcnRhbnQgYW5kIHJlZHVkYW50IGZlYXR1cmVzIHRvd2FyZHMgemVyby4KCgpgYGB7ciBzYXZlL2xvYWQgbGFzc28gYm9vdHN0cmFwIG1vZGVsIHBlcmZvcm1hbmNlLCBpbmNsdWRlPUZBTFNFfQoKYnNfbGFzc29fcGVyZiA9IE5VTEwKICAKIyBnZXQgZGF0YSBmaWxlcwpic19sYXNzb19wZXJmID0gbG9hZF9mcm9tX2Nzdihic19sYXNzb19wZXJmLCByZXN1bHRzX2RpcmVjdG9yeSwgbm9fdGFza3MpCgoKYGBgCgpgYGB7ciBldmFsdWF0ZSBib290c3RyYXAgbW9kZWwgcGVyZm9ybWFuY2UgbGFzc28sIHdhcm5pbmc9RkFMU0UsIG1lc3NhZ2U9RkFMU0V9Cm1lYW5fYnNfbGFzc29fcGVyZiA8LSBic19sYXNzb19wZXJmICU+JQogIHN1bW1hcmlzZV9pZihpcy5udW1lcmljLCBtZWFuLCBuYS5ybT1UUlVFKSAlPiUgCiAgbXV0YXRlKG1vZGVsPSdic19sYXNzbycpICU+JQogIGRwbHlyOjpzZWxlY3QobW9kZWwsIGV2ZXJ5dGhpbmcoKSkKCm1lYW5fYnNfbGFzc29fcGVyZgpgYGAKCiMjIyBGZWF0dXJlIGltcG9ydGFuY2VzOiBMQVNTTwojIyMjIENvZWZmaWNpZW50LWJhc2VkIHZhcmlhYmxlIGltcG9ydGFuY2UKCmBgYHtyIHNhdmUvbG9hZCBsYXNzbyBib290c3RyYXAgbW9kZWwgbWRpLCBpbmNsdWRlPUZBTFNFfQoKYnNfbGFzc29fbWRpID0gTlVMTAogIAojIGdldCBkYXRhIGZpbGVzCmJzX2xhc3NvX21kaSA9IGxvYWRfZnJvbV9jc3YoYnNfbGFzc29fbWRpLCByZXN1bHRzX2RpcmVjdG9yeSwgbm9fdGFza3MpCgpgYGAKCmBgYHtyIHNob3cgYnMgbGFzc28gbm9ybWFsaXplZCBjb2VmZmljaWVudHN9CmJvb3RfbGFzc29fbWRpIDwtIGxpc3QoYnNfbGFzc29fbWRpKSAlPiUKICBnZXRfbWVkaWFuX3BsYWNlbWVudCh1c2VfYmFzZV92YXIgPSBUUlVFKSAlPiUKICBhZGRfYXR0cmlidXRlX25hbWVzKCdwcmVkaWN0b3InLCBmdWxsX2RhdGFzZXQpICU+JQogIGRwbHlyOjpzZWxlY3QocHJlZGljdG9yLCBhdHRfbmFtZSwgb3ZlcmFsbF9yYW5rKQoKaGVhZChib290X2xhc3NvX21kaSwgMjApCmBgYAoKYGBge3IgZmlnLndpZHRoID0gMTAsIGZpZy5oZWlnaHQgPSAxMn0KcGxvdF9wbGFjZW1lbnRfYm94cGxvdChsaXN0KGJzX2xhc3NvX21kaSkpCmBgYAoKIyMjIyBQZXJtdXRhdGlvbiBpbXBvcnRhbmNlCgpgYGB7ciBjb21wdXRlL3NhdmUgb3IgbG9hZCBicyBsYXNzbyBtb2RlbCBwZXJtdXRhdGlvbiwgaW5jbHVkZT1GQUxTRX0KCmJzX2xhc3NvX3Blcm1fcGx0ID0gTlVMTAogIAojIGdldCBkYXRhIGZpbGVzCmJzX2xhc3NvX3Blcm1fcGx0ID0gbG9hZF9mcm9tX2Nzdihic19sYXNzb19wZXJtX3BsdCwgcmVzdWx0c19kaXJlY3RvcnksIG5vX3Rhc2tzKQoKYGBgCgoKYGBge3IgYWdncmVnYXRlIGxhc3NvIHBlcm11dGF0aW9ucyBhbmQgZ2V0IG1ldHJpY3N9CmJzX2xhc3NvX3Blcm0gPC0gYnNfbGFzc29fcGVybV9wbHQgJT4lCiAgZ2V0X3Blcm11dGVfcGxhY2VtZW50KG1ldHJpY19vaT1tZXQpICU+JSAjc2V0IGluIHJhbmRvbSBmb3Jlc3Qgc2VjdGlvbgogIGFkZF9hdHRyaWJ1dGVfbmFtZXMoJ3ByZWRpY3RvcicsIGZ1bGxfZGF0YXNldCkgJT4lCiAgZHBseXI6OnNlbGVjdChwcmVkaWN0b3IsIGV2ZXJ5dGhpbmcoKSkKCmhlYWQoYnNfbGFzc29fcGVybSwgMjApCmBgYAoKYGBge3IgcGxvdCBsYXNzbyBwZXJtdXRhdGlvbiwgZmlnLndpZHRoID0gMTAsIGZpZy5oZWlnaHQgPSAxMn0KcGxvdF9wZXJtdXRlX3Zhcl9pbXAoYnNfbGFzc29fcGVybSwgbWV0cmljID0gbWV0KQpgYGAKCiMjIyMgQ29lZmZpY2llbnQgdnMuIFBlcm11dGF0aW9uIGltcG9ydGFuY2UKTm93LCB3ZSBjb21wYXJlIHRoZSBmZWF0dXJlIGltcG9ydGFuY2VzIGdlbmVyYXRlZCBieSB0aGUgdHdvIGRpZmZlcmVudCBhcHByb2FjaGVzLiAgVGhlIHRyYWRpdGlvbmFsIG1ldGhvZCBvZiBldmFsdWF0aW5nIGZlYXR1cmUgaW1wb3J0YW5jZSBmb3IgcmVncmVzc2lvbiBtZXRob2RzIGlzIHRocm91Z2ggYW5hbHlzaXMgb2YgdGhlIGNvZWZmaWNpZW50cy4KYGBge3IgZmlnLndpZHRoID0gMTYsIGZpZy5oZWlnaHQgPSAxNH0KY2JpbmQoYm9vdF9sYXNzb19tZGlbMToyMCxdLCBkcGx5cjo6c2VsZWN0KGJzX2xhc3NvX3Blcm1bMToyMCxdLCAtbWV0KSkKYGBgCgojIyBDb21wYXJpc29uOiBNb2RlbCBUeXBlIE1lYW4gUGVyZm9ybWFuY2UKVGhlIGZvbGxvd2luZyB0YWJsZSBjb21wYXJlcyB0aGUgbWVhbiBwZXJmb3JtYW5jZSBvZiBib290c3RyYXBwZWQgcmFuZG9tIGZvcmVzdHMgdG8gdGhlIG1lYW4gcGVyZm9ybWFuY2Ugb2YgYm9vdHN0cmFwcGVkIExBU1NPIG1ldGhvZHMuCmBgYHtyIGNvbXBhcmUgYnMgbWVhbiBwZXJmb3JtYW5jZX0KYnNfY29tcF9wZXJmcyA8LSBiaW5kX3Jvd3MobWVhbl9ic19yZl9wZXJmLCBtZWFuX2JzX2xhc3NvX3BlcmYpIApic19jb21wX3BlcmZzCmBgYAoKIyMgQ29tcGFyaXNvbjogTW9kZWwgVHlwZSBGZWF0dXJlIEltcG9ydGFuY2UKSGVyZSwgd2UgbG9vayBhdCB0aGUgYWdncmVnYXRlZCByZXN1bHRzIG9mIHRoZSBib290c3RyYXBwZWQgcHJlZGljdG9ycyBhbmQgY29tcGFyZSB0aGUgbW9kZWxzIGdlbmVyYXRlZCB0byBlYWNoIG90aGVyLgpgYGB7ciBjb21wcmUgYnMgZmVhdHVyZSBpbXBvcnRhbmNlfQpqb2luZWRfcmVzdWx0cyA8LSBic19yZl9wZXJtICU+JQogIGRwbHlyOjpzZWxlY3QoLW1ldCkgJT4lCiAgZnVsbF9qb2luKGRwbHlyOjpzZWxlY3QoYnNfbGFzc29fcGVybSwgLW1ldCksIGJ5PWMoInByZWRpY3RvciIsICJhdHRfbmFtZSIpLCBzdWZmaXg9YygnLnJmJywgJy5sYXNzbycpKSAlPiUKICBtdXRhdGUobWVhbl9yYW5rID0gKG92ZXJhbGxfcmFuay5yZitvdmVyYWxsX3JhbmsubGFzc28pLzIpICU+JQogIGFycmFuZ2UobWVhbl9yYW5rKQoKaGVhZChqb2luZWRfcmVzdWx0cywgMjApCmBgYAoKVGhlIGZvbGxvd2luZyB2aXN1YWxpemF0aW9uIHByb3ZpZGVzIHRoZSBpbnR1aXRpb24gYWJvdXQgdGhlIGRpZmZlcmVuY2VzIGluIHRoZSByYW5raW5ncyBiZXR3ZWVuIG1vZGVsIHR5cGVzLiAgVGhleSdyZSBvcmRlcmVkIGJ5IHRoZSBvdmVyYWxsIG1lYW4gaW1wb3J0YW5jZSwgYW5kIGZvciBhIGdpdmVuIHZhcmlhYmxlLCB0aGUgZGlmZmVyZW5jZXMgaW4gcmFuayBhcmUgc2hvd24uCmBgYHtyIGJzIGZlYXR1cmUgaW1wb3J0YW5jZSBwbG90bHksIGZpZy53aWR0aCA9IDE2LCBmaWcuaGVpZ2h0ID0gMTR9CiMgQ29tcGFyaXNvbiBvZiB0b3BfbiBmZWF0dXJlcwpqb2luZWRfcmVzdWx0cyAlPiUKICBjb21wYXJlX2ZlYXR1cmVfc2VsZWN0KGludGVyYWN0aXZlID0gVFJVRSwKICAgICAgICAgICAgICAgICAgICAgICAgIHRvcF9uID0gMTAwLAogICAgICAgICAgICAgICAgICAgICAgICAgb3BhY2l0eSA9IDAuNTAsCiAgICAgICAgICAgICAgICAgICAgICAgICBwbG90X3RpdGxlID0gIlBlcm11dGF0aW9uIEltcG9ydGFuY2Ugb2YgUHJlZGljdG9ycyBieSBNb2RlbCIpCmBgYAoKCiMgR2VuZXJhdGlvbiBvZiBmaW5hbCBtb2RlbCB7LnRhYnNldCAudGFic2V0LWZhZGUgLnRhYnNldC1waWxsc30KCiMjIFJGIG1vZGVsCkluIHRoaXMgc3RlcCwgd2UgYnVpbGQgdGhlIGZpbmFsIG1vZGVsIGZvciB0aGUgcmFuZG9tIGZvcmVzdC4gIFdlIHVzZSBzbGlnaHRseSBtb3JlIHZhbHVlcyBpbiBvcmRlciB0byBjb21lIHVwIHdpdGggdGhlIGJlc3QgbW9kZWwsIGtlZXBpbmcgaW4gbWluZCB0aGUgbnVtYmVyIG9mIGNvbWJpbmF0aW9ucyB0aGF0IGFyZSByZXF1aXJlZCB0byBydW4gdG8gZXZhbHVhdGUgdGhlIGdyaWQuCgpgYGB7ciBzYXZlL2xvYWQgZmluYWwgcmYgbW9kZWx9CgpmaW5hbF9yZl9wZXJmID0gTlVMTAogIAojIHJlYWQgZmlsZSBhcnJheQpmaW5hbF9yZl9wZXJmcyA9IGxvYWRfZnJvbV9jc3YoZmluYWxfcmZfcGVyZiwgcmVzdWx0c19kaXJlY3RvcnksIG5vX3Rhc2tzKQoKIyBnZXQgdGhlIGluZGV4IG9mIHRoZSBiZXN0IHBlcmZvcm1hbmNlIGFuZCBiZXN0IHBlcmZvcm1hbmNlCmZpbmFsX3JmbW9kZWxfaW5kIDwtIHdoaWNoLm1heChkcGx5cjo6c2VsZWN0KGZpbmFsX3JmX3BlcmZzLCBtZXQpICU+JSBwdWxsKG1ldCkpCmZpbmFsX3JmX3BlcmYgPC0gZmluYWxfcmZfcGVyZnMgJT4lIHNsaWNlKGZpbmFsX3JmbW9kZWxfaW5kKQoKIyBsb2FkIHRoZSBtb2RlbCBvZiBpbnRlcmVzdApmaW5hbF9tb2RlbF9yZiA8LSBsb2FkX2Jlc3RfbW9kZWwoJ2ZpbmFsX3JmX21vZGVsJywgcmVzdWx0c19kaXJlY3RvcnksIGZpbmFsX3JmbW9kZWxfaW5kKQoKYGBgCgojIyMgUGVyZm9ybWFuY2UKVGhlIGZpbmFsIHJhbmRvbSBmb3Jlc3QgcGVyZm9ybWFuY2UgbWV0cmljcyBhcmUgc2hvd24gYmVsb3c6CmBgYHtyIHJmIGZpbmFsIG1vZGVsIHBlcmZvcm1hbmNlfQojIHNob3cgbW9kZWwgZmluYWwgcGVyZm9ybWFuY2UKcHJpbnQoZmluYWxfcmZfcGVyZikKYGBgCgojIyMgRmVhdHVyZXM6IHBlcm11dGF0aW9uIGltcG9ydGFuY2UKCmBgYHtyIGNvbXB1dGUvc2F2ZSBvciBsb2FkIGZpbmFsIHJmIG1vZGVsIHBlcm11dGF0aW9ufQoKZmluYWxfcmZfcGVybV9wbHQgPSBOVUxMCiAgCiMgcmVhZCBiZXN0IHBlcm0gcGx0CmZpbmFsX3JmX3Blcm1fcGx0ID0gbG9hZF9mcm9tX2NzdihmaW5hbF9yZl9wZXJtX3BsdCwgcmVzdWx0c19kaXJlY3RvcnksIGJlc3RfaW5kPWZpbmFsX3JmbW9kZWxfaW5kKQoKYGBgCgoKYGBge3IgZ2V0IG1ldHJpY3MgZm9yIGZpbmFsIHJmfQpmaW5hbF9yZl9wZXJtIDwtIGZpbmFsX3JmX3Blcm1fcGx0ICU+JQogIGdldF9wZXJtdXRlX3BsYWNlbWVudChtZXRyaWNfb2k9bWV0KSAlPiUKICBhZGRfYXR0cmlidXRlX25hbWVzKCdwcmVkaWN0b3InLCBmdWxsX2RhdGFzZXQpICU+JQogIGRwbHlyOjpzZWxlY3QocHJlZGljdG9yLCBldmVyeXRoaW5nKCkpCgpoZWFkKGZpbmFsX3JmX3Blcm0sIDIwKQpgYGAKCmBgYHtyIHBsb3QgcmYgZmluYWwgcGVybXV0YXRpb24sIGZpZy53aWR0aCA9IDEwLCBmaWcuaGVpZ2h0ID0gMTJ9CnBsb3RfcGVybXV0ZV92YXJfaW1wKGZpbmFsX3JmX3Blcm0sIG1ldHJpYyA9IG1ldCkKYGBgCiMjIyBDb21wYXJpc29uIHdpdGggYm9vdHN0cmFwIHJlc3VsdHMKVGhpcyBzZWN0aW9uIGludmVzdGlnYXRlcyB0aGUgZGlmZmVyZW5jZXMgaW4gdGhlIGJvb3RzdHJhcCByZXN1bHRzIHZzIHRoZSBmZWF0dXJlcyBnZW5lcmF0ZWQgZnJvbSB0aGUgcmFuZG9tIGZvcmVzdCBmaW5hbCBtb2RlbC4gIFRoZSBmb2xsb3dpbmcgdGFibGUgc2hvd3MgdGhlIG92ZXJhbGwgZGlmZmVyZW5jZXMgaW4gcmFuay4KCmBgYHtyIGNvbXBhcmUgcmYgZmluYWwgd2l0aCBib290c3RyYXB9CnJmX2pvaW5lZF9yZXN1bHRzIDwtIGZpbmFsX3JmX3Blcm0gJT4lCiAgZHBseXI6OnNlbGVjdCgtbWV0KSAlPiUKICBmdWxsX2pvaW4oZHBseXI6OnNlbGVjdChic19yZl9wZXJtLCAtbWV0KSwgYnk9YygicHJlZGljdG9yIiwgImF0dF9uYW1lIiksIHN1ZmZpeD1jKCcuZmluYWwnLCAnLmJvb3RzdHJhcCcpKSAlPiUKICBtdXRhdGUobWVhbl9yYW5rID0gKG92ZXJhbGxfcmFuay5maW5hbCArIG92ZXJhbGxfcmFuay5ib290c3RyYXApLzIpICU+JQogIGFycmFuZ2UobWVhbl9yYW5rKQoKaGVhZChyZl9qb2luZWRfcmVzdWx0cywgMjApCmBgYAoKVGhlIGZvbGxvd2luZyBwbG90IHByb3ZpZGVzIHZpc3VhbGl6YXRpb25zIGZvciB0aGUgZGlmZmVyZW5jZSBpbiB0aGUgZmluYWwgbW9kZWwgcmFua2luZ3MgdnMgdGhlIGJvb3RzdHJhcC4KCmBgYHtyIHBsb3QgZmluYWwgcmYgZmVhdHVyZXMgcGxvdGx5LCBmaWcuaGVpZ2h0PTE0LCBmaWcud2lkdGg9MTZ9CiMgQ29tcGFyaXNvbiBvZiB0b3BfbiBmZWF0dXJlcwpyZl9qb2luZWRfcmVzdWx0cyAlPiUKICBjb21wYXJlX2ZlYXR1cmVfc2VsZWN0KHNlbF9jb2xzID0gYygib3ZlcmFsbF9yYW5rLmZpbmFsIiwgIm92ZXJhbGxfcmFuay5ib290c3RyYXAiKSwKICAgIGludGVyYWN0aXZlID0gVFJVRSwKICAgIHRvcF9uID0gMTAwLAogICAgb3BhY2l0eSA9IDAuNTAsCiAgICBwbG90X3RpdGxlID0gIlBlcm11dGF0aW9uIEltcG9ydGFuY2Ugb2YgUHJlZGljdG9yczogRmluYWwgdnMuIEJvb3RzdHJhcCIpCmBgYAoKIyMgTEFTU08gbW9kZWwKTm93LCB3ZSBjcmVhdGUgdGhlIGZpbmFsIG1vZGVsIGZvciBMQVNTTy4gIFRoZXJlIGlzIG5vIHN1YnN0YW50aWFsIGRpZmZlcmVuY2UgYmV0d2VlbiB0aGlzIG1ldGhvZCBhbmQgdGhlIGJvb3RzdHJhcCBtZXRob2RzLCBvdGhlciB0aGFuIHRoZSBkYXRhIHVwb24gd2hpY2ggdGhlIG1vZGVsIGlzIGJlaW5nIGJ1aWx0LgoKYGBge3Igc2F2ZS9sb2FkIGZpbmFsIGxhc3NvIG1vZGVsfQoKZmluYWxfbGFzc29fcGVyZiA9IE5VTEwKICAKIyByZWFkIGZpbGUgYXJyYXkKZmluYWxfbGFzc29fcGVyZnMgPSBsb2FkX2Zyb21fY3N2KGZpbmFsX2xhc3NvX3BlcmYsIHJlc3VsdHNfZGlyZWN0b3J5LCBub190YXNrcykKCiMgZ2V0IHRoZSBpbmRleCBvZiB0aGUgYmVzdCBwZXJmb3JtYW5jZSBhbmQgYmVzdCBwZXJmb3JtYW5jZQpmaW5hbF9sYXNzb21vZGVsX2luZCA8LSB3aGljaC5tYXgoZHBseXI6OnNlbGVjdChmaW5hbF9sYXNzb19wZXJmcywgbWV0KSAlPiUgcHVsbChtZXQpKQpmaW5hbF9sYXNzb19wZXJmIDwtIGZpbmFsX2xhc3NvX3BlcmZzICU+JSBzbGljZShmaW5hbF9sYXNzb21vZGVsX2luZCkKCiMgbG9hZCB0aGUgbW9kZWwgb2YgaW50ZXJlc3QKZmluYWxfbW9kZWxfbGFzc28gPC0gbG9hZF9iZXN0X21vZGVsKCdmaW5hbF9sYXNzb19tb2RlbCcsIHJlc3VsdHNfZGlyZWN0b3J5LCBmaW5hbF9sYXNzb21vZGVsX2luZCkKCmBgYAoKVGhlIGZpbmFsIExBU1NPIHBlcmZvcm1hbmNlIG1ldHJpY3MgYXJlIHNob3duIGJlbG93OgpgYGB7ciBsYXNzbyBmaW5hbCBtb2RlbCBwZXJmb3JtYW5jZX0KIyBzaG93IG1vZGVsIGZpbmFsIHBlcmZvcm1hbmNlCnByaW50KGZpbmFsX2xhc3NvX3BlcmYpCmBgYAoKIyMjIEZlYXR1cmVzOiBwZXJtdXRhdGlvbiBpbXBvcnRhbmNlCgpgYGB7ciBjb21wdXRlL3NhdmUgb3IgbG9hZCBmaW5hbCBsYXNzbyBtb2RlbCBwZXJtdXRhdGlvbn0KCmZpbmFsX2xhc3NvX3Blcm1fcGx0ID0gTlVMTAoKI2xvYWQgYmVzdCBpbmRleCBwZXJtdXRhdGlvbiBmcm9tIGZpbGUgIApmaW5hbF9sYXNzb19wZXJtX3BsdCA9IGxvYWRfZnJvbV9jc3YoZmluYWxfbGFzc29fcGVybV9wbHQsIHJlc3VsdHNfZGlyZWN0b3J5LCBiZXN0X2luZD1maW5hbF9sYXNzb21vZGVsX2luZCkKCgpgYGAKCmBgYHtyIGdldCBwZXJtdXRlIGZlYXR1cmVzIGZvciBmaW5hbCBsYXNzb30KZmluYWxfbGFzc29fcGVybSA8LSBmaW5hbF9sYXNzb19wZXJtX3BsdCAlPiUKICBnZXRfcGVybXV0ZV9wbGFjZW1lbnQobWV0cmljX29pPW1ldCkgJT4lCiAgYWRkX2F0dHJpYnV0ZV9uYW1lcygncHJlZGljdG9yJywgZnVsbF9kYXRhc2V0KSAlPiUKICBkcGx5cjo6c2VsZWN0KHByZWRpY3RvciwgZXZlcnl0aGluZygpKQoKaGVhZChmaW5hbF9sYXNzb19wZXJtLCAyMCkKYGBgCgpgYGB7ciBwbG90IGxhc3NvIGZpbmFsIHBlcm11dGF0aW9uLCBmaWcud2lkdGggPSAxMCwgZmlnLmhlaWdodCA9IDEyfQpwbG90X3Blcm11dGVfdmFyX2ltcChmaW5hbF9sYXNzb19wZXJtLCBtZXRyaWMgPSBtZXQpCmBgYAojIyMgQ29tcGFyaXNvbiB3aXRoIGJvb3RzdHJhcCByZXN1bHRzClRoaXMgc2VjdGlvbiBpbnZlc3RpZ2F0ZXMgdGhlIGRpZmZlcmVuY2VzIGluIHRoZSBib290c3RyYXAgcmVzdWx0cyB2cyB0aGUgZmVhdHVyZXMgZ2VuZXJhdGVkIGZyb20gdGhlIExBU1NPIGZpbmFsIG1vZGVsLiAgVGhlIGZvbGxvd2luZyB0YWJsZSBzaG93cyB0aGUgb3ZlcmFsbCBkaWZmZXJlbmNlcyBpbiByYW5rLgoKYGBge3IgY29tcGFyZSBsYXNzbyBmaW5hbCB3aXRoIGJvb3RzdHJhcH0KbGFzc29fam9pbmVkX3Jlc3VsdHMgPC0gZmluYWxfbGFzc29fcGVybSAlPiUKICBkcGx5cjo6c2VsZWN0KC1tZXQpICU+JQogIGZ1bGxfam9pbihkcGx5cjo6c2VsZWN0KGJzX2xhc3NvX3Blcm0sIC1tZXQpLCBieT1jKCJwcmVkaWN0b3IiLCAiYXR0X25hbWUiKSwgc3VmZml4PWMoJy5maW5hbCcsICcuYm9vdHN0cmFwJykpICU+JQogIG11dGF0ZShtZWFuX3JhbmsgPSAob3ZlcmFsbF9yYW5rLmZpbmFsICsgb3ZlcmFsbF9yYW5rLmJvb3RzdHJhcCkvMikgJT4lCiAgYXJyYW5nZShtZWFuX3JhbmspCgpoZWFkKGxhc3NvX2pvaW5lZF9yZXN1bHRzLCAyMCkKYGBgCgpUaGUgZm9sbG93aW5nIHBsb3QgcHJvdmlkZXMgdmlzdWFsaXphdGlvbnMgZm9yIHRoZSBkaWZmZXJlbmNlIGluIHRoZSBmaW5hbCBtb2RlbCByYW5raW5ncyB2cyB0aGUgYm9vdHN0cmFwLgoKYGBge3IgcGxvdCBsYXNzbyBmZWF0dXJlIGNvbXBhcmlzb24sIGZpZy53aWR0aCA9IDE2LCBmaWcuaGVpZ2h0ID0gMTR9CiMgQ29tcGFyaXNvbiBvZiB0b3BfbiBmZWF0dXJlcwpsYXNzb19qb2luZWRfcmVzdWx0cyAlPiUKICBjb21wYXJlX2ZlYXR1cmVfc2VsZWN0KHNlbF9jb2xzID0gYygib3ZlcmFsbF9yYW5rLmZpbmFsIiwgIm92ZXJhbGxfcmFuay5ib290c3RyYXAiKSwKICAgIGludGVyYWN0aXZlID0gVFJVRSwKICAgIHRvcF9uID0gMTAwLAogICAgb3BhY2l0eSA9IDAuNTAsCiAgICBwbG90X3RpdGxlID0gIlBlcm11dGF0aW9uIEltcG9ydGFuY2Ugb2YgUHJlZGljdG9yczogRmluYWwgdnMuIEJvb3RzdHJhcCIpCmBgYAoKIyMgQ29tcGFyaXNvbjogRmluYWwgbW9kZWwgZmVhdHVyZXMKSGVyZSwgd2UgY29tcGFyZSB0aGUgZmVhdHVyZXMgZ2VuZXJhdGVkIGJ5IHRoZSBwZXJtdXRhdGlvbiBpbXBvcnRhbmNlIGJldHdlZW4gdGhlIHR3byBmaW5hbCBtb2RlbHMuCgpgYGB7ciBjb21wYXJlIGxhc3NvIHZzIHJhbmRvbSBmb3Jlc3QgZmVhdHVyZXMgZmluYWx9CnJmX2xhc3NvX2ZpbmFsX2pvaW5lZF9yZXN1bHRzIDwtIGZpbmFsX3JmX3Blcm0gJT4lCiAgZHBseXI6OnNlbGVjdCgtbWV0KSAlPiUKICBmdWxsX2pvaW4oZHBseXI6OnNlbGVjdChmaW5hbF9sYXNzb19wZXJtLCAtbWV0KSwgYnk9YygicHJlZGljdG9yIiwgImF0dF9uYW1lIiksIHN1ZmZpeD1jKCcucmYnLCAnLmxhc3NvJykpICU+JQogIG11dGF0ZShtZWFuX3JhbmsgPSAob3ZlcmFsbF9yYW5rLnJmK292ZXJhbGxfcmFuay5sYXNzbykvMikgJT4lCiAgYXJyYW5nZShtZWFuX3JhbmspCgpoZWFkKHJmX2xhc3NvX2ZpbmFsX2pvaW5lZF9yZXN1bHRzLCAyMCkKYGBgCgpUaGUgZm9sbG93aW5nIHZpc3VhbGl6YXRpb24gcHJvdmlkZXMgdGhlIGludHVpdGlvbiBhYm91dCB0aGUgZGlmZmVyZW5jZXMgaW4gdGhlIHJhbmtpbmdzIGJldHdlZW4gdGhlIGZpbmFsIG1vZGVsIHR5cGVzLiAgVGhleSdyZSBvcmRlcmVkIGJ5IHRoZSBvdmVyYWxsIG1lYW4gaW1wb3J0YW5jZSwgYW5kIGZvciBhIGdpdmVuIHZhcmlhYmxlLCB0aGUgZGlmZmVyZW5jZXMgaW4gcmFuayBhcmUgc2hvd24uCgpgYGB7ciBwbG90bHkgbGFzc28gdiByYW5kb20gZm9yZXN0IGZlYXR1cmVzfQojIENvbXBhcmlzb24gb2YgdG9wX24gZmVhdHVyZXMKcmZfbGFzc29fZmluYWxfam9pbmVkX3Jlc3VsdHMgJT4lCiAgY29tcGFyZV9mZWF0dXJlX3NlbGVjdChzZWxfY29scyA9IGMoIm92ZXJhbGxfcmFuay5yZiIsICJvdmVyYWxsX3JhbmsubGFzc28iKSwKICAgIGludGVyYWN0aXZlID0gVFJVRSwKICAgIHRvcF9uID0gMTAwLAogICAgb3BhY2l0eSA9IDAuNTAsCiAgICBwbG90X3RpdGxlID0gIlBlcm11dGF0aW9uIEltcG9ydGFuY2Ugb2YgUHJlZGljdG9yczogUmFuZG9tIEZvcmVzdCB2cyBMYXNzbyIpCmBgYAoKIyMgQ29tcGFyaXNvbjogRmluYWwgbW9kZWwgcGVyZm9ybWFuY2UKV2l0aCB0aGUgZmluYWwgbW9kZWxzIGdlbmVyYXRlZCwgd2UncmUgbm93IGFibGUgdG8gY29tcGFyZSB0aGVpciBwZXJmb3JtYW5jZSBtZXRyaWNzLgpgYGB7ciBmaW5hbCBtb2RlbCBjb21wYXJpc29uLCBmaWcud2lkdGggPSAxMCwgZmlnLmhlaWdodCA9IDEyfQojIENvbXBhcmlzb24gb2YgcGVyZm9ybWFuY2UgbWV0cmljcwp2YWxpZF9wZXJmIDwtIGdldF9tZXRyaWNfc2V0X2Zyb21fcGVyZnMocGVyZl9saXN0ID0gbGlzdChmaW5hbF9yZl9wZXJmLCBmaW5hbF9sYXNzb19wZXJmKSkgJT4lCiAgbXV0YXRlKG1vZGVsID0gYygncmYnLCAnbGFzc28nKSkKCnRlc3RpbmdfcGVyZiA8LSBnZXRfbWV0cmljX3NldF9mcm9tX21vZGVscyh0ZXN0aW5nX2RmLCBsaXN0KGZpbmFsX21vZGVsX3JmLCBmaW5hbF9tb2RlbF9sYXNzbyksIG91dD1vdXRjb21lKSAlPiUKICBtdXRhdGUobW9kZWwgPSBjKCdyZicsICdsYXNzbycpKQpgYGAKCioqVmFsaWRhdGlvbiBhbmQgc2VsZWN0aW9uLioqClRoZSBmb2xsb3dpbmcgdGFibGUgc2hvd3MgdGhlIGNvbXBhcmlzb24gYmV0d2VlbiBtb2RlbHMgaW4gdGVybXMgb2YgdGhlIHZhbGlkYXRpb24gc2V0LiAgV2UgY2FuIHNlbGVjdCBvdXIgZmluYWwgbW9kZWwgYmFzZWQgb24gdGhlIGJlc3QgcGVyZm9ybWluZyBtb2RlbCBhY2NvcmRpbmcgdG8gdGhlIG1ldHJpYy4KYGBge3J9CnByaW50KHZhbGlkX3BlcmYpCmBgYAoKKipUZXN0aW5nIHBlcmZvcm1hbmNlLioqClRoZSBmb2xsb3dpbmcgc2hvd3MgdGhlIHBlcmZvcm1hbmNlIG9mIGJvdGggdGhlIG1vZGVscyBvbiB0aGUgdGVzdCBzZXQuICBOb3RlIHRoYXQgYWx0aG91Z2ggd2UgZG9uJ3QgdXNlIHRoaXMgdGVzdCBzZXQgdG8gZXZhbHVhdGUgdGhlIGZpbmFsIG1vZGVscywgd2UgY2FuIHN0aWxsIHNlZSBob3cgb3VyIHNlbGVjdGVkIG1ldGhvZCB3b3VsZCBoYXZlIHBlcmZvcm1lZC4KYGBge3J9CnByaW50KHRlc3RpbmdfcGVyZikKYGBgCgpUaGUgZm9sbG93aW5nIHBsb3RzIHNob3cgYSBjb21wYXJpc29uIGJldHdlZW4gdGhlIHBlcmZvcm1hbmNlIG9mIHRoZSBtb2RlbHMgb24gdGhlIHZhbGlkYXRpb24gYW5kIHRlc3Qgc2V0cy4gIEFnYWluLCB3ZSBkb24ndCBjaG9vc2UgdGhlIG1vZGVsIGJhc2VkIG9uIHRoZSB0ZXN0IHNldCwgYnV0IGN1cmlvc2l0eSBkaWN0YXRlcyB0aGF0IHdlIHZpZXcgdGhpcyBwZXJmb3JtYW5jZS4KCmBgYHtyIGZpZy53aWR0aCA9IDEwLCBmaWcuaGVpZ2h0ID0gNn0KIyBTaG93IHBsb3RzIHNpZGUgYnkgc2lkZQptZXRyaWNzX29mX2ludGVyZXN0ID0gYygnbW9kZWwnLCAnYWNjdXJhY3knLCAnYmFsX2FjY3VyYWN5JywgJ21wY2UnLCAnc2VucycsICdzcGVjJywgJ3BwdicsICducHYnLCAncHJfYXVjJywgJ3JvY19hdWMnKQp2YWxpZF9wbHQgPC0gcGxvdF9tZXRyaWNfc2V0KGRwbHlyOjpzZWxlY3QodmFsaWRfcGVyZiwgYWxsX29mKG1ldHJpY3Nfb2ZfaW50ZXJlc3QpKSwgcGxvdF90aXRsZSA9ICJNb2RlbCBjb21wYXJpc29uIGZvciB2YWxpZGF0aW9uIHNldCIpCnRlc3RfcGx0IDwtIHBsb3RfbWV0cmljX3NldChkcGx5cjo6c2VsZWN0KHRlc3RpbmdfcGVyZiwgYWxsX29mKG1ldHJpY3Nfb2ZfaW50ZXJlc3QpKSwgcGxvdF90aXRsZSA9ICJNb2RlbCBjb21wYXJpc29uIGZvciB0ZXN0aW5nIHNldCIpCmdyaWRFeHRyYTo6Z3JpZC5hcnJhbmdlKGdyaWRFeHRyYTo6YXJyYW5nZUdyb2IodmFsaWRfcGx0LCB0ZXN0X3BsdCwgbmNvbD0yLCBucm93PTEpKQpgYGAKIyBPdGhlciBjbGVhbnVwCmBgYHtyIHNodXRkb3duIGgybyBpbnN0YW5jZX0KaDJvLnNodXRkb3duKEZBTFNFKQpgYGA=